
Proceedings 

Selected Research and Development Papers 
Presented at The Annual Convention of the Association for 
Educational Communications and Technology - Volume 1 



Sponsored by the Research and Theory Division 
Louisville, KY 


Editor: Michael Simonson 


Nova Southeastern University, North Miami Beach, Florida 



2012 Annual Proceedings - Louisville: Volumes 1 


Volume 1: Selected Research and Development Papers 


Presented at 

The Annual Convention of the Association for Educational Communications and Technology 
Sponsored by the Research and Theory Division 
Louisville, KY 
2012 


Editor 

Michael Simonson 
Professor 

Instructional Technology and Distance Education 
Nova Southeastern University 
Fischler Graduate School of Education and Human Services 
North Miami Beach, FL 



Preface 


For the thirty-fifth year, the Research and Theory Division of the Association for Educational 
Communications and Technology ( AECT) is sponsoring the publication of these Proceedings. Papers 
published in this volume were presented at the annual AECT Convention in Louisville, KY. A limited 
quantity of these Proceedings were printed and sold in both hardcopy and electronic versions. Volumes 1 
and 2 are available through the Educational Resources Clearinghouse (ERIC) System. Proceedings volumes 
are available to members at AECT.ORG. 

The Proceedings of AECT's Convention are published in two volumes. Volume #1 contains papers dealing 
primarily with research and development topics. Papers dealing with the practice of instructional 
technology including instruction and training issues are contained in Volume #2. This year, both volumes 
are included in one document. 

REFEREEING PROCESS: Papers selected for presentation at the AECT Convention and included in these 
Proceedings were subjected to a reviewing process. All references to authorship were removed from 
proposals before they were submitted to referees for review. Approximately sixty percent of the 
manuscripts submitted for consideration were selected for presentation at the convention and for 
publication in these Proceedings. The papers contained in this document represent some of the most current 
thinking in educational communications and technology. 


Michael R. Simonson 
Editor 
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Expert Instructional Designer Voices: Leadership Competencies Critical to Global Practice and Quality Online 
Learning Designs 


Abstract 

This paper offers qualitative evidence that leadership competencies and characteristics activate quality global 
learning technologies, although those competencies remain mostly unrecognized as having relevance to instructional 
design (ID). In a study designed to explore current perceptions of what is needed to improve the quality of online 
courses, the voices of an international group of instructional designers were captured and analyzed for common 
themes of significance to the ID community. Analysis of the discourse engaged in during in-depth interviews 
provided a framework of leadership characteristics that were positively associated with high quality pedagogies — 
strategy, vision, personality (interpersonal skills), productivity, emotional/psychological strength, values, and duties. 
Lrom these themes, a model of leadership is presented, followed by suggestions for urgently needed streams of 
research into the critical issue of ID leadership for quality. 

Introduction 

In spite of on-going advances in 21st century instructional technologies, web-based higher academic 
pedagogies continue to demonstrate a lack of quality (Means, Toyama, Murphy, Bakia, & Jones, 2009). The 
perception of lower academic standards is emphasized by an on-going distrust in online courses by a majority of US 
educators (Allen & Seaman, 2012; Seaman, 2009). Lrom an international perspective, Daniel (2007) referred to 
online pedagogies in India as “mostly rubbish” (Affordability section, para. 15), while Uysal and Kuzu (2009) 
reported a problem in Turkey of no online standards in existence. Online learning offers a flexible option for those 
with various restrictions, including geographical access and financial constraints. It has been posited that the current 
economic challenges for students seeking to compete on a global scale with an advanced education has contributed 
to a continual rise in online enrollments (Allen & Seaman, 2010; Robinson, 2009). However, the persistent state of 
less than stellar online courses undermines the efficacy of educational opportunities afforded by the internet. 

In light of perceptions of lower quality, the practice of designing products for online learning has been 
called into question in terms of the competencies and leadership of instructional designers (Beaudoin, 2007; Kowch, 
2009; Naidu, 2007; Reeves, Herrington, & Oliver, 2004; Sims & Koszalka, 2008) needed to improve the situation. 
This notion aligns with the business and trade industries where improvement of products has been inextricably 
linked to leadership. Lor example, Steve Jobs, after an earlier departure, returned to Apple in 1997 and led an 
industry with re-envisioned ways of communicating information (Isaacson, 201 1). Who can deny that his leadership 
resulted in exemplary products? Consequently, a study was undertaken that would explore a potential connection 
between the quality of ID products and an instructional designer's leadership competencies. 

The study’s findings emerged from an international panel of expert practitioners’ lived experiences, 
conveyed through perceptions of a practice that blurs the lines of standards usually understood for ID (Association 
of Educational Communications and Technology (AECT), 2012; Larson & Lockee, 2009). The participants 
specified competencies for designing online pedagogies with the added dimension of applied personal leadership 
characteristics deemed critical to high quality designs — Competencies, Attributes, and Duties. In addition, specific 
instructional strategy components known to predict engagement (NSSE, 2008) and learner satisfaction (Sims & 
Stork, 2007) framed a model for leading the online design process forward — strategy, vision, personality 
(interpersonal skills), productivity, emotional/psychological strength, values, and duties. As a result of these 
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findings, the goal of the study was satisfied: to determine whether leadership from instructional designers would 
have a significant impact on the vision many share for quality online higher educational products. 

Background and Context 

As an outcome of studying the ID process and its value to online learning, certain questions percolated 
which called for answers. What is needed to improve quality in online courses? What competencies may be critical 
to design effective pedagogies for a changing student body with diverse needs? What characteristics of ID are 
lacking to meet global socio-educational demands? Although standard ID competencies (AECT, 2012; IBSTPI, 
2000) are well established for best practices, it was noticed that leadership competencies were nearly absent from 
the prescriptions. Upon further exploration, the topic of leadership as a characteristic of instructional designers was 
just as scarce (Kowch, 2009). What was found in the literature were suggestions that instructional designers lead 
educational transformations by embracing a need to change inappropriate or misapplied practices (Beaudoin, 2007; 
Kowch, 2009; Naidu, 2007; Reeves, et al., 2004; Sims, 2006). The notion corresponds with Smith (in Fullan & 
Scott, 2009) who declared a quiet crisis in higher education from dependence on a model that “flies in the face of 
what we know about how people learn, the opportunities that technology presents to transform the educational 
enterprise and our historic record of failure with a rapidly diversifying population” (p. 20). In addition, Fullan and 
Scott (2009) observed a lack of leadership competency and experience in those expected to be change agents of 
education. The concerns amplified reports of ineffective and irrelevant design practices such as text-based lectures 
loaded into a course room, practice components, multiple choice quizzes and tests, a deck of slides for learning 
enhancement, a term paper (Naidu, 2007) — and not much more. 

Questioning the design approach of adapting traditional course designs to the online environment was 
important because studies were beginning to show that people actually learn differently interacting with a computer 
than with a human interface (Dede, Dieterle, Clarke, Jass-Ketelhut, & Nelson, 2007; Spiro & DeSchryver, 2009). 
On-screen text and slideshows may not have engaged the learner, yet various multimedia affordances were showing 
promise to capture and hold attention (Thompson et al., 2011). Dede et al.’s (2007) study found positive learning 
outcomes from students connecting and learning in new ways — with computers, essentially a new mode of learning 
(Beaudoin, 2007). Fater, Dede’s (201 1) explorations took him into mobile learning and the potential of engaging in 
learning in new ways with cell phones. The challenge for educators and producers of pedagogical materials is to 
progress along the same continuum as the new paradigm of digital learning. 

To explore the problem of quality in online course designs, a qualitative research study purposed to identify 
the critical ID competencies needed to produce effective designs for modern learning. Acting on the assumption 
that there may be a connection between leadership competencies — or characteristics in general — and an approach to 
the design process, an inquiry method of studying expert instructional design professionals in the context of practice 
was proposed and undertaken. 


Theoretical Framework 

A literature review conducted for purposes of grounding the study presented in this paper found that 
leadership theory encompasses copious philosophies, competencies, traits, attributes, attitudes, positions, and roles, 
which have been applied to a multitude of professions (Zenger & Folkman, 2009). To narrow the focus of the study, 
competency theory provided a relevant lens of exploration and was defined as behavioral demonstrations of 
knowledge, skill, and ability (Dooley, Lindner, Telg, Irani, Moore, & Lundy, 2007); this was relevant to the study of 
ID practice, one that demands multiple competencies (Larson & Lockee, 2009). During the literature analysis, it 
was discovered that both leadership and competency components were often interchanged with attributes and 
character traits. To mitigate obfuscation, and to gain a consensus of meaning for each term that would inform the 
study, the review resourced from a multi-disciplinary body of literary work. The cross-disciplinary approach was 
necessary, as the ID literature revealed a paucity of conceptualizations of leadership in personal practice or of its 
impact on the field and its products (Kowch, 2009; Spillane et al., 2004). Thus, a synthesis of theories framed the 
study being presented with multivariate characteristics of leadership categorized under competencies and attributes. 

Leadership Competencies 

A major predictor of work performance is competency, which enables the accomplishment of a desired or 
prescribed task (Dooley et al., 2007). Competency is a general category characterized by multiple components, of 
which strategy, vision, personality (interpersonal skills), and productivity are defined below. 
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First, leaders are competent in developing a strategy that will include proactive plans for the future (Scott, 
Coates, & Anderson, 2008). In other words, they reflect on methods to prevent problems rather than wait for them 
to happen. The Scott et al. (2008) study revealed that participants perceived leaders as engaging others in the 
process as well as collaborating for best possible solutions to, not only current but, unforeseen problems and 
challenges. Furthermore, leaders know where to go for the answers or the knowledge to create answers; they make 
the right connections in a network of colleagues and technology (Siemens, 2004; Sims, 2006). 

A leader is commonly known to possess vision and. as such, is recognized as a visionary (Floward & 
Wellins, 2008). A leader also serves as a “steward of the vision" according to Sackney and Mergel (2007, p. 94). 
This person not only sees the vision, lives the vision through decisions, conveys the vision to others, enlivens others 
to the vision, promotes the vision but, at the same time, encourages others to share in the vision ( Kouzes & Posner, 
2007). Leaders as visionaries recognize innovation, forward-thinking, uniqueness, and "respond creatively to world 
conditions and the current state of their own society” (Greenleaf, 1977, p. 321). Scott et al. (2008) regarded this 
characteristic as having a capacity to see the big picture and to “read and respond to a continuously and rapidly 
changing external environment” (p. 1 1), a notion that may also be interpreted as possessing global competency 
(Reimers, 2009). 

Leaders convey personality within the dynamics of human interactions. The notion resonated with a 
pervasive claim in literature that communication is the linchpin of all successful leadership (Sergiovanni & 

Corbally, 1984). Sergiovanni and Corbally (1984) defined interpersonal competence as an essential force of 
leadership. Moreover, demonstrations of genuineness stem from active communication. Kouzes and Posner (2007) 
described leaders as caring, confident, and respectful to others. In contrast, Howard and Wellins (2008) cited the 
primary cause of failure by a large percentage of organizational leaders was lacking in interpersonal skills. 

Leaders have been described as those who understand how to work hard to achieve results; in other words, 
competency is evident in their productivity. A study of leadership roles in virtual teams showed that learners 
preferred a producer (Chen, Wu, Yang, & Tsou, 2008), a leader who gets things done, over a visionary. In this 
capacity, leaders are expected to do what they say (Argyris & Schon, 1992) by converting words to action. In brief, 
results are produced by skillful leaders who have been trained in specific capacities, and work until the job is done. 
When taken together, the competencies listed also describe the work of a designer or an ID team, and, consequently, 
provide a glimpse of how leadership may link to ID practice. 

Leadership Attributes 

The category of attributes included emotional/psychological strength and values. Leadership is having 
confidence and displaying strength in diverse ways (Kirkpatrick & Locke, 1991). Leaders are logical, make good 
choices, and think rationally, according to Kepner and Tregoe (1997). Thus, the various ways of conveying 
emotional strength and psychological depth are measured in how one thinks through and responds to challenges 
(Scott et al., 2008). 

Finally, an array of values describe leadership, including one operating from conscience (Wolumbwa, 
Avolio, Gardner, Wernsing, & Peterson, 2008) and personal convictions, and being grounded in moral and ethical 
principles (Covey, 1989; Senge, 2006). With significance to one of this study's key assumptions, that leadership is a 
predictor of quality course designs, Wolumbwa et al. (2008) put forth the notion that moral leadership is a predictor 
of “relevant organization outcomes” (p. 91). Therefore, enhanced leadership competencies are proposed herein as 
factors in activating and maintaining quality, globally-accessed learning designs. 

Methodology 


Research Design 

For exploration into the problem of quality in online pedagogical course designs, a phenomenological study 
engaged a modified Delphi-style approach to invite expert instructional designers with a common interest in 
resolving a shared problem within a defined social setting (Gelo et al., 2008; Wenger, 1998) to participate in a 
narrative inquiry study. Understanding that experience impacts decision-making (Ertmer et al., 2008), experts were 
specified in order to discover what drives the critical decisions that impact pedagogical quality of online outputs. 
Following purposive selection from 610 members of the AECT that were solicited through email, seven respondents 
met the qualifications of at least seven years’ experience in ID, with three of those in online design (those selected 
averaged over 16 years’ experience); and out of the seven, six completed the study. The group included both male 
and female participants from US and Canadian universities. Sources of data included interview transcripts from 
research conversations, emails, personal documents, and institution-generated student evaluations. For analysis of 
findings, constant comparison analysis (Glaser & Strauss, 1977) was used to examine data until congruency was 
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found in the units of meaning identified for thematic analysis. Subsequently, the interview data were corroborated 
for accuracy and relevance to personal documents — online course designs created by the participants — which were 
analyzed with a modified version of the Quality Matters (QM) rubric® for testing the quality of online courses. 

Data Instrumentation and Collection 

As the researcher and main instrument of collection, I asked a series of questions that were each an 
expansion on the general question of what leadership competencies were deemed critical to effective online ID 
course development. To establish credibility and to guide the study, a research protocol was developed to organize 
the data collection procedure. The protocol called for three rounds of in-depth interviews during which the 
participants were asked the same 15 semi -structured, open-ended questions. However, certain questions inspired 
important digressive thoughts; consequently, probing questions were added for eliciting clarification and deeper 
meaning. Throughout the study, ID voices were captured in the context of daily practice in their respective 
institutions. Therefore, all data and citations used in the following analysis are confined to these settings, although 
the ideas recorded may resonate with others in similar situated contexts. In this way, the participating IDs were 
asked to inform the study from a social constructivist perspective, to construct meaning for a shared practice 
(Wenger, 1998). Experts often voice common experiences which, when made known, become triggers of change in 
a community of practice (Campbell, et al., 2009). 

Several tests of reliability were included in the research protocol. Because of the intimate nature of 
narrative research and to minimize influence on the participant’s responses, the researcher practiced bracketing 
(Moustakas, 1994) — which is the setting aside of one’s personal beliefs and ideas. This is not always an easy task 
when involved in dialogue, as a researcher will naturally insert personality, interest, and investment in the topic into 
the interview dynamics. To the extent possible, though, bracketing was accomplished through active listening and 
by taking copious research notes, notes which later provided for reflexivity (Kaplan, 2004). For example, one 
journal recorded perceptions of participants’ emotional states, attitudes, and responsiveness to the questions or to the 
interviewer. The other journal recorded the research process, researcher emotions and attitudes, and the challenges 
of doing research. Further, in an attempt to reach group consensus on the competencies and characteristics named, 
inter-rater reliability was established by comparing participants’ answers, then feeding those back to each participant 
for critical comments. At the conclusion of the study, all methodologies, tests, and findings were corroborated by an 
external qualitative design expert. Although the study was small in terms of participants, the rich data from a group 
of experts in ID was, at the least, a great conversation re-starter of what has been, and is, considered an important 
characteristic for designers of online products, leadership. 

Analysis of Findings 

A post hoc analysis of transcripts from over 14 hours of recorded narrative relied on NVivo9 software for a 
constant comparative analysis technique. Phrases (units of meaning) from the transcribed data were labeled with 
descriptors found in the business, organizational, and educational literature. Initially, the numerous phrases were 
organized under the general categories of leadership characteristics: Competencies, Attributes, and Duties. 

However, further distillation generated the themes: strategy, vision, personality (interpersonal skills), productivity, 
emotional strength, values, and a duty to mentor or to pass on knowledge. The iterative process resulted in some 
phrases being reallocated; although, at times, the technique generated additional themes and continued to build 
theory for the study. A list of the themes, along with minimal sample excerpts from the data (abbreviated in most 
cases), are displayed in Table 1 to convey how phrases were organized into themes. 
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Table 1. Themes Extracted from Participants in a Qualitative, In-Depth Interview Study of ID Experts 


Theme 

Excerpt 

Participant 

Strategy 

Meet goals set by client: learning objectives, strategies, and personalized 

P4 


Allow customer to be part of decisions 

P5 


Collaborative team-building 
Interactivity for digital learner 

PI 


Offer client informed options 

P2 


Synthesize best learning and ID theories; include real-world tasks in design 

P3 

Vision 

Stay current 

P5 


Backup designs with research 

PI 


Partners with other educators to accept new approach 

P3 

Productivity 

Quality control by team testing each element 

PI 


Manage time, resources; meet deadlines, prioritize, produce good product 

P6 

Personality 

Satisfy customer through communication and provide solutions 

P5 

(Interpersonal Skills) 

They ( the team) have fun 

PI 


Provide expectations, build others up 

P6 

Emotional/Psychologi 

Accept other’s ideas and go with, work with ideas of others (letting go of 

P2 

cal Strength 

positional privilege) 


Values 

Respect is foundation of leadership in design process 

P2 


Accept consequences of decisions, self-leadership, self-regulation, self- 
management, self-discipline 

P6 

Duties (Mentor) 

Model collaboration (to faculty, team) 

P6 


Since the context and focus of the study centered in the participants' daily experiences, an effort was made 
during the analysis phase to link the perceptions to practice. Various conceptions and perceptions comprise one’s 
daily reality in terms of meanings and interpretations attached to the world in which one operates (Pratt, cited in 
Konings et al., 2005), as well as govern decisions in the work environment (Stein, Shephard, & Harris, 2009). To 
associate the leadership competencies and characteristics named as critical to quality course designs, participants’ 
sample documents (online course designs) were evaluated using a modified Quality Matters (QM) rubric®. The 
quality of the documents was confirmed by the positive results for course structure, including relevant and effective 
instructional strategies and components. A second analysis observed an alignment between measureable objectives, 
relevant assessments, instructional strategies, and authentic activities, a construct considered essential to quality 
academic designs (Bernard, Abrami, Borokhovski, Wade, Tamin, Surkes, & Bethel, 2009; Sims, 2011). Finally, an 
analysis of student evaluations accompanying the artifacts found three major quality determinants present: 
opportunities for engagement (NSSE, 2008), activities with learner-centeredness at the core (Sims & Stork, 2007), 
and overall student satisfaction with the courses. 

The results not only identified aspects of practice deemed to be critical to its practitioners; they also 
suggested a relationship of leadership to higher quality pedagogies. As accuracy was considered paramount, general 
first impressions by the researcher of the participants’ narratives, along with results of initial data analyses, were 
reported back to the participants in both verbal and written form for comments and elaboration. The corroborative 
step was completed before the third and final round of interviews and lent additional reliability to the study, as a 
more complete context of the research study’s findings was illuminated by each individual participant’s situational 
perspective (Seidman, 2006). To capture the essence of their unique contributions to the study, a profile of each 
participant was developed and is located in Exhibit A. 
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A Model of Leadership for ID 


Overall, the findings showed that a leadership mindset is positively associated with high quality work 
products when certain characteristics are in operation, which formed a model of leadership for ID. The major 
categories of the model — Competencies, Attributes, and Duties — as well as the themes listed in Table 1, are 
synthesized and displayed in graphic form in Figure 1 . Subsequently, the model is more fully explicated in this 
section. 



Figure 1. Model of leadership for instructional design (ID) developed from major themes of research study. 

Strategy 

When the participants discussed various aspects of strategy, two distinct subthemes were identified: overall 
design process strategy and instructional strategy for the design relevant to digital environments. The panel 
members articulated a leadership mindset to make (a) strategic decisions for the institution to align epistemological 
and administrative approaches with online learner demands, (b) strategic decisions for the design team to research 
best options for providing strict alignment of objectives to assessments, content, and activities, and (c) strategic 
decisions for the design structure to afford engagement with context-relevant learning flexible enough to adapt 
quickly to advancing technologies. 

Design Process Strategy 

The importance of the leadership competency of strategy for the organization and design team resonated 
with Scott et al. (2008) who implied that leaders are competent in developing strategic, proactive plans for the 
future. In other words, leaders are adept at engaging others in the process to collaborate for best possible solutions 
to current and unforeseen problems and challenges. Dooley et al. (2007) concluded, this type of competency is 
evident during the creation and implementation of courses and in the critical decisions underpin high quality, 
effective, and relevant designs. In this capacity, leadership is essential to the building and functioning of the team 
for an efficacious design and development effort. Participant 6 stated this position on decision-making. 

Leadership really deals more with the types of decision that one has to make and the way that one makes 
and executes these decisions... decision-making skill is definitely one of those leadership competencies... as 
well as willingness to accept the consequences of your decisions. . .you take responsibility for that and you 
don’t shift it to someone else. 
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Another aspect of strategy is connectedness. The views of participants in this study were consistent with 
those (Siemens, 2004; Sims, 2006) who posited that leaders make the right connections in a network of colleagues 
and technology for expertly accomplishing the task at hand or anticipated. Likewise, Scott et al. (2008) posited that 
leaders know where to go for the answers or knowledge to create the answers to questions of design. 

Collaboration with others in the field, as well as with those from other disciplines, adds a dimension of 
connectedness to practice from which springs inspiration for creating innovative and quality courses. Sims (2009) 
described a proactive mindset of collaboration and connectedness with each design stakeholder to ensure quality 
learning products as outcomes to ID efforts. Participantl stated, 

I think it’s the leaders who can create and will create collaborative teams made up of people with diverse 
perspectives and talents, skills, and expertise... the ones who will be developing the engaging, motivating, 
interactive, and effective learning environments for today’s learners. 

The study also found that a critical factor in producing quality courses is staying current with research in 
technologies and emerging learning theories. Connectedness and collaboration are not dissociative. One participant 
gave an illustration of an online research writing course he had written that resulted in a reported satisfying outcome 
for the learners. It required tailoring the course to the research he had done on current best practices for online 
learning and from collaborating with others who had already envisioned and/or designed a course like this. He also 
relied on media experts for the latest methods of presenting the course content; these were strategic leadership 
actions that resulted in an effective course. Summarizing this aspect of the strategy theme. Participant 3 expressed, 

. . .you have to have a knowledge of the latest learning theories and the latest instructional design theories in 
order to be a leader. You also have to be able to implement those theories, individually and in concert with 
each other, to achieve best practice, instructional strategies, or instructional strategy design. 

At the same time, online delivery requires letting go of outdated practices for effective learning. Participant 1 
recognized this need, 

We've known ADDIE forever, we’ve known this model forever; but, the online models are different 
because it’s a different delivery modality. So just get the most current research and information you can on 
what is best practices. 

Design Structure Strategy 

The second type of strategy linked to leadership competency was described as the ability to specify 
appropriate — research-based and contextual — instructional strategies with relevant learning components — to the 
student and to the environment, and having those align with the intended learning outcome. A depiction of this 
concept is conveyed in Figure 2 and articulated by Participant P6, 

We would tend to focus on, as far as basic axioms of instructional design, the one that objectives are 
measurable and that the objectives are aligned with the assessments, and the objectives and the assessments 
are aligned with the content. . .allowing you to meet the objectives. 

A second and related issue is the development of the instructional strategy proposed for the design. Figure 
3 is an aggregated list of the components identified by the study group as paramount to a quality online course; 
authentic tasks, interaction, learner-controlled tasks, problem-solving, theory-based instruction, and values-based 
instruction. Together, the components provide a foundation of engagement and learner satisfaction through a 
contextualized and personalized learning approach posited by Sims and Stork (2007) as essential for effective online 
learning. 
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Figure 2. Components of a well-designed Figure 3. Components of a well-designed 

online course structure. online course instructional strategy. 

Going beyond the identification of strategic components in general, the expert designer’s experiences 
paralleled examples of innovative approaches in the literature, such as social networking and mobile technologies 
(Velletsianos & Miller, 2008). Their perceptions underscored Reimer’s (2009) conclusions that instructional 
designers demonstrate competence — distributive, collaborative, and participatory leadership — by specifying 
nontraditional resources where appropriate for, in his study, an international student body. For example. The 
University of the People (www.universityopeople.org) requires its designers to select open source (free) textbooks, 
applications, and journals to expand access and affordability for their free online program. Participant 2 expressed 
her strategy as having the flexibility to see a future outcome through a unique point of view and to expand her own 
knowledge of design components and their potential as having, as she stated, 

. . .the ability to recognize and work within the concepts of different and better [emphases added]. 

It is this kind of innovative thinking that demonstrates strategic leadership on the part of the instructional designers 
with an eye on the future. 

Vision 

Zenger and Folkman (2009) studied the effectiveness of leaders from a cross section of organizations and 
identified the characteristic of strategy in operation, which was summarized as, “to translate organizations 
[punctuation missing in original] vision and objectives into challenging and meaningful goals for others” (p. 70). 

The assumption is that business — and educational — institutions first develop a vision before a strategy. Likewise, 
instructional designers translate the vision and objectives of a client for an educational goal into a course design. 
From this perspective Participant 6 remarked, “I take information and I translate it into an environment that 
facilitates learning.” During such pursuit, the ID infuses a measure of his or her own vision for what may be the 
appropriate structure, strategy, affordance, and learning components for a high quality learning experience. 
Participant 6 interpreted this competency as, “the ability to see the big picture”. With a similar perspective, 
Participant 1 likened the visionary role of an ID to leading a sports team, 

. . .it’s having this quarterbacking skill, being able to read and see things two or three tasks ahead so that 
you. . . can foresee possible issues or challenges that are going to come up and try to get those taken care of 
before your team is at that level. 
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Second, a global view for the field and for education in general was viewed by one panel member as being 
aware of “contemporary'’ and “emerging technologies.” Going beyond awareness to action. Participant 2 stated 
vision is. 


. . .the ability to lead in the design. . .implementation. . .development of the new things that come along. 

Instructional designers may envision and design for future trends in delivering education afforded by social 
networking technologies (Ashbaugh, in press). The findings in this present study confirmed that instructional 
designers demonstrate vision by specifying nontraditional resources, such as wikis (Baggio, 2008). However, 
having a vision for new pedagogies carries a responsibility to investigate its potential. One participant shared a story 
of experimentation with a wiki before she had gained a thorough understanding of how one works. Her 
embarrassment over not having learned enough about the technology was obvious as she told her story of “a 
disaster” in front of a classroom full of students. The lesson learned was that, while freely exploring innovation, 
designers need to resist prescribing new technologies before doing as much research as possible. 

Personality (Interpersonal Skills) 

For the design team, communication and meaningful dynamics are essential to a quality outcome and 
emanate from a leader’s personality. Larson and Lockee (2009) considered interpersonal communication within 
diverse settings as vital; although, how to approach others in the intimate settings instructional designers often 
operate within may require a characteristic of personal leadership. The study participants related that instructional 
designers listen carefully to arguments by the team or clients and articulate decisions from a practical, firm position. 
Interpersonal skills afford designers of online pedagogies an opportunity to exhibit leadership by offering academic 
and relevant arguments for the design, strategic, and team decisions while taking responsibility for the final 
outcomes. This view of leadership in action was expressed by Participant 1 as being, 

. . .able to communicate technological possibilities, letting people know what is possible with technology. . . 
being able to be quite frank with them. 

Participant 6 shared his experience with leading in creating positive team dynamics, 

I need to provide an atmosphere, a working atmosphere, that is not intimidating and that helps them feel 
that they can be creative, and that they can inject their ideas and criticisms in areas that they see needs 
improvement, so that they can participate with this. 

Another participant stressed the need for patience and partnership by working with a client to help them see a better 
way, through the lens of a professional, while acknowledging the client’s and the advising subject matter expert’s 
(SME) unique contributions. 

From a synthesis of the participants’ perceptions on personality, it was apparent that the typical view of 
communication considered essential to leadership (Howard & Wellins, 2008) — ability to articulate — may not 
provide a broad enough understanding. Communication assumes various forms which stem from personality — 
interpersonal skills and attitude, ability to convey ideas and to practice reciprocity, and respect and honesty. Clearly, 
in this study it was shown that an ID brings a unique personality to a design with a potential for leading in its quality 
outcome through each dynamic of interpersonal encounters. 

Productivity 

The study group named most of the known competencies of productivity associated with a good ID process 
such as: “meet deliverables”, “lead in implementation”, “time management”, “know strengths of team”, as well as 
motivating skillset improvement and keeping costs down. Often, the range of tasks involved in developing 
academic courses within client time and budget constraints (Moore & Kearsley, 2005) demands leading in hard 
work as well as finishing the job, in spite of limitations by institutions bent on following what Campbell et al. (2005) 
deemed cost-recovery models. In the context of preferring cost over quality. Participant 4 personally witnessed a 
less expensive approach to practice of relying on template-based design strategies. Considered an unproductive 
practice, she remarked, 

I understand why they were doing what they were doing. . .but they could never reach real high quality 
doing what they were doing. 
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Ostensibly, there is an inherent quality of leadership that emanates from an inner desire for meeting goals, 
for getting the job done, and for completion without sacrificing relationships. Participant 6 viewed the motivational 
aspect of leadership as “set [ting] the course” while another Participant 1 summarized, 

. . . [it’ s] using your inner relational skills to build positive, collaborative, cohesive teams to meet your 
deliverables. . .plus you have to know how to motivate them if you’re going to get all this done. . .and 
helping them honor and respect each other’s skills and expertise. 

From an end results perspective. Participant 1 added. 

The lead instructional designer sets the tone for the team. If he or she is proficient, the end product will 
reflect that. 

Emotional/Psychological Strength 

A leader possesses or gains a measure of emotional or psychological strength, which is often displayed in 
intentional decisions made from self-confidence and strength of purpose; although, this is not always an easy task 
when confronting inflexible institutions and faculty with entrenched epistemological ideas(Campbell et al., 2009). 
Conversely, participants of this study described experiences in which an inner strength was activated to lay aside 
personal preference in lieu of honoring a colleague or client’s theories. This attribute was seen in the previous 
theme of strategy when a participant recognized an outdated practice and used a better approach. Maxwell (2007) 
considered leaders psychologically strong when they operate on the offensive and not the defensive, with openness 
to other ideas, opinions, and criticisms. On the other hand, Campbell et al. (2009) found instructional designers in 
moral conflict with client epistemologies and often without a sense of power to make changes. To this end, the 
study participants cited having “a lot of patience” and “being able to work with it [emphasis added].” Clearly, 
instructional designers are called upon to make complex, emotional decisions within a shifting learning environment 
while maintaining a rational approach ( Kepner & Tregoe, 2007); and this study suggests, they will need leadership 
strengths to succeed. 

V alues 

One area of obvious concern for instructional designers that is operating from a set of personal values — for 
the entire spectrum of stakeholders although particularly for the learner. Care is given to provide for the vulnerable, 
the disabled, and a multi-cultural learning community. One particular participant in this study stressed his lived out 
values and perceived personal leadership in this way: 

I think that, for a leader, it really involves who you are even more than what you do. . .in the field of 
instructional design, I guess I’m looking at more character traits than competencies. . .1 would probably still 
hold honesty as the most important. . .it’s about honesty and integrity 

The personal perspective of leaders with values and the significant influence on design practice is echoed 
repeatedly in the literature and has been conceptualized as an individual having an inherent moral purpose (Fullan, 
2001). Moreover, the Campbell et al. (2009) study revealed deeply-held values in instructional designers who 
considered their practice a moral one with important social responsibility and influence, who “prefer[ed] to practice 
within a zone of moral coherence” (p. 660). 

Duties 

The theme of ID duties was discussed by participants mainly in the context of mentorship. With a sense of 
obligation to pass on knowledge gained through experience, instructional designers interpret the value of course 
designs to the client and to others through verbal and nonverbal interactions. In the process, there is a natural 
tendency to act as a mentor or instructor in the more complex nuances of design. As one Participant 4stated about 
her experience with clients. 

You are educating at the same time that you are doing the design. . .you have to double check on that all the 
time, that people actually understand what it is you’re doing. 
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Adding a layered view to the mentoring process. Participant 2 articulated the need for acting in multiple capacities in 
her role as an ID with faculty oversight. 

So, mentoring the junior people just in the basics of instructional design and how it works, sometimes on an 
individual level, sometimes they’re working in the context of the larger project, so in the project of the 
team. . .and sometimes I have teams of instructional designers, so it may be your journeyman instructional 
designer who is actually him or herself mentoring these juniors. . .to help them mentor the juniors. 

A perception of diminished status by instructional designers in a Rogers, Graham, and Mayes (2007) study 
implicated a field tasked with educating the world from a powerless position. Campbell et al. (2009) described 
study subjects’ views on the designer's status as a “technician that primarily implements techniques and principles” 
(p. 661). The study participants (Campbell et al., 2009) described feelings of low respect and held perceptions of 
negative impact on their professional status and influence. It was argued that instructional designers suffered from 
an historical view of leadership as one of a positional role. 

However, it is increasingly evident that individuals in any role benefit from a leadership perspective 
(Gressick & Derry, 2010; Kowch, 2009). Beaudoin (2007) supported the notion of distributed and multiple 
leadership roles as important to the future of instructional technology (p. 520). More than this, the ID demands 
regard for the earned role of leader (DeBlois, 2005). In an interview study of faculty and practicing instructional 
designers DeBlois (2005) found a majority shared a perspective that includes a change agent and visionary role for 
leading the academy on what and how to change for a fluid educational world. Most significant to the study 
conclusions offered in this paper, the participants in that study linked success with embracing the leadership 
attributes needed for advancing the field. Therefore, instructional designers of higher academic online pedagogies 
are challenged to update the competencies and approaches that underpin exemplary practice. 

Discussion 

This study has shown that multiple competencies and leadership characteristics combine to create a quality 
product. Zenger and Folkman (2009) coded thousands of responses to their longitudinal survey study and arranged 
the findings into a model of five expressions of exemplary leadership — character, leading organizational change, 
focus on results, personal capability, and interpersonal skills. With close resemblance to their work, what was 
critical for leading in quality design practice was summarized by PI, 

Leadership competencies are, again, having a process, having a process that works; pulling together people, 
making sure that the interrelationship skills are there and working; knowing who to contact for help; again, 
using clear communication skills to convey the purpose and timelines, keeping them on the timelines; and. 
trying to build a collaborative team who have buy-in and ownership of this course. 

In contrast to that participant’s more global perspective. Participant 3 stressed his view of technical skills as critical 
to quality. He stated, 

Unless you can implement best practice in instructional strategies, the online instruction, basically, remains 
back in the dark ages of instruction. . .Looking at online course quality is figuring out which factors matter 
the most. And so when everything is said and done, leadership is about finding those factors. . . 

Participant 2 summarized the study’s findings with these thoughts, 

1 do think that the leadership qualities are integral to the course design, to the student learning, to the 
outcomes that you’re going to get, to future transfer. . .not only in the subject matter content that you are 
doing, but also transfer in the instructional design principles and the educational precepts that are really 
important in higher education everywhere. 

With this response, the significance and importance of the study was distilled with the learner at the apex. 

From the evidence gathered in this study, it is posited that practicing instructional designers are capable of 
adopting and operationalizing leadership skills, abilities, and attitudes critical for improving course designs that 
equip future world citizens and leaders. As one participant expressed, “it’s having a mindset.” That mindset is 
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broad and demands a new way of thinking. A new way of thinking or a new approach is often linked to those who 
lead, and just as often to the development of new or better products. 

As online education progresses, the burden of research is continuing to show that course structures and 
materials written for one type of learning situation need to be different than when learning in another — and since 
most instructional designers in practice today were trained for traditional requirements, they may need to be re- 
tooled for creating relevant and quality online products. In other words, instructional designers need a reorientation 
in thinking about designing for an evolutionary change in educational deliveries. This is because the literature 
increasingly called for a response to the demands of a new paradigm of learners for an instructional design vision 
that will encourage global collaboration among institutions (Durdu, Yalabik, & Cagiltay, 2009) and cultures 
(Reimers, 2009), through best-fit models and relevant curricula for quality online learning. Reimers (2009) 
contended global competency is essential for designers of online initiatives privileged with the responsibility of 
training the world’s learners in respectful awareness of and value for other cultures. 

Arguably, leadership is a controversial concept for the typical designer who has been trained for a 
supportive role (Campbell et al., 2009) with very little potential for leading on a personal or positional level 
(Gressick & Derry, 2010), let alone lead in a new way of learning. To help quell apprehensions of an important, 
reborn concept, this paper intended to illuminate the voices of expert instructional designers who found leadership to 
be a critical component of exemplary ID practice. 

Conclusion and Future Research 

The seldom discussed notion of leadership for daily ID practice ( Kowch, 2009) posited in this paper was 
intended to be an extension of a conversation started by leaders in the field during the last decade (Beaudoin, 2007; 
Naidu, 2007; and others) who called for leadership recognition for an effective ID practice. The experts whose 
voices are heard in this paper augmented the concerns of their predecessors. Together, the calls underscored the 
motivational purpose of this study: to identify and understand what will enhance the quality of learning events for 
students from any culture who are, or hope to be, engaged in e-learning. It has been the aim of this study to present 
multiple streams of thought as to how leadership drives and delivers a quality product for online environments and 
all learners. The voices of expert instructional designers contributing to this study created a sense of urgency for 
creating new research streams for infusing leadership competency training into advanced degree programs, 
extending inquiry into practitioner’s perceptions of leadership's place in daily practice, and broadening research 
variables such as gender, ethnic, and geographic effects. 

There remains a need for methodical research to confirm current views on what role leadership plays in the 
product quality of instructional designers. The critical nature of improving the structures for learning outcomes was 
highlighted by Participant 3 who warned that if quality does not become the focus of attention and dissemination in 
the field of online instructional design, “the majority of the instruction is going to remain inferior.” One remedy to 
this indictment was suggested earlier by Hannum (2009) who admonished researchers to rigorously explore 
“variables that directly influence outcomes” (p. 173) of the online environment. Still others shared the need for 
continuing domain-specific research on strategies required in a modern learning environment (Bollettino & 
Bruderlein, 2008; Hong & Sullivan, 2009). Conducting a perceptual study of Canadian graduate students, Webber 
and Robertson (2004) found a need for greater cross-cultural understanding of the learner's educational contexts, as 
well as a need for international thinking about leadership. Finally, with vision in mind, Reimers (2009) promoted 
emergence of grassroots efforts to organize, define, and promote global competency throughout curricula — an on- 
going challenge for leaders in instructional design. 

To further this discussion, course designers and others are urged to pay attention to the powerful voices of 
experts in the field guiding this study for how increased leadership characteristics in practice will move the online 
education agenda forward toward more excellent products. It is further hoped that the conversation begun by IDs in 
the previous decades will gain renewed attention to the more critical elements of leadership and its role in the quality 
of global learning events. 
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Exhibit A 

Profiles of Study Participants 


Participant 1 (PI) 

A full-time professor who instructs at a United States university with significant online course offerings, PI 
leads a teacher training department from both an instructional technology and educational leadership background. 
While bringing 12 years of experience to the study. Pi's extensive knowledge in issues of quality course designs 
added to the findings of this study during three interviews. Her contributions of personal documents were 
incorporated into the analysis of relevant artefacts and student evaluations. 

While filtering leadership through a lens of team leader, this participant emphasized interpersonal skills and 
knowledge of team skills in that role. PI focused on a leader's need for personal knowledge, skills, expertise, and 
proficiency. Design leadership competencies perceived by her included knowledge of current research on best 
practices and team-building. PI was concerned with increasing student learning by aligning course strategies and 
activities with course and program outcomes. In addition, PI expressed a belief that leadership decisions influence 
designs through knowledgeable choices for affordances such as appropriate technology and texts, as well as reliance 
on student feedback for improvement. Moreover, the experienced practitioner attributed a team-building, 
collaborative process to a successful practice of quality online course design. 

Participant 2 (P2) 

P2 is a full-time professor with teaching, training, and research duties for a Canadian university’s 
educational technology department. The participant reported 22 years of experience in the field and brought her 
expertise into this study through three interviews, personal artefacts, and student evaluations. 

While filtering leadership through a lens of a team leader role, P2 emphasized leadership as a means to an 
end: creating effective learning (happy clients) through team collaboration and growth. She focused on a leader's 
ability to lead projects, giving clients informed options, and creating innovative, team-based designs. Design 
leadership competencies perceived by P2 included recognition of what is “different and better” and respectfully 
conveying the validity of instructional design principles underlying the recommendations offered. Additionally, P2 
was concerned with increasing student learning by advocating activity theory, mediation with tools, constructivist 
strategies, and authentic activities for metacognitive learning. With conviction, P2 acknowledged a leader’s 
decisions influence designs through reliance on student involvement, students as “co-builders” of contextual, 
situational course topics and assessments; ultimately, leaders impact student satisfaction in these ways. 

Participant 3 (P3) 

With over seven 7 of experience practicing in the instructional design field, P3 added knowledge to this 
study from multiple perspectives and background in the e-learning industry. While completing a PhD in 
instructional technology (at a United States university), P3’s extensive research in assessing quality of course 
designs lent valuable input during three interviews and from personal artifacts and evaluations. 

While filtering leadership through a lens of expertise in and application of instructional design theories, P3 
emphasized influencing others in improving quality of online instruction in that role. He focused on a leader's 
ability to implement latest instructional design strategies for best practice. Design leadership competencies 
perceived by P3 included knowledge of current research, partnership with developers and instructors, and influence 
on many from a position of expertise and from evidence-based knowledge of best practices and approaches. 
Furthermore, P3 was concerned with increasing student learning by affording better and current technology-based 
strategies and inclusion of real-world tasks in the design affordances. 

Participant 4 (P4) 

P4 claimed a semi-retired status as an expert instructional designer with 20 years of experience in the field. 
As an adjunct online professor with a technical university, instructed doctoral students, and was a frequent presenter 
at educational technology conferences worldwide. Although withdrawing from the study after two interviews, her 
valuable contributions to the study were included in the findings. 

While filtering leadership through a lens of modelling behavior through patience in collaborating with the 
stakeholders, P4 emphasized understanding and aligning with institutional and students’ educational goals in 
improving quality of online instruction. She works toward a course design that meets the needs of all learners, no 
matter their learning style or mode. Design leadership competencies perceived by P4 included knowledge of current 
research, as well as conference attendance and presentations as ways of being a “life-long learner” in the field. P4 
was concerned with increasing student learning by affording good student interaction strategies in the course room. 
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Participant 5 (P5) 

A director of a United States university learning center providing faculty development in instructional 
technology, P5 was working to improve his practice by seeking a PhD in the field. Quality of instructional design 
was the focus of his work, making him a particularly qualified candidate for this study. Although study participation 
was limited to the first interview due to scheduling conflicts, P5’s important insights were included in the findings 
for RQ1. 

Participant 6 (P6) 

As Dean of Online Studies and Distinguished Lecturer of Graduate Studies for a United States university, 
P6 contributed significant empirical data from his 20 years of experience in the field of educational technology and 
design. The scholar presented compelling views on the study topic during three interviews and from personal 
documents with student evaluations. 

While filtering leadership through a lens of both technical and foundational skills in instructional design 
systems, P6 emphasized honesty, interpersonal skills, and self-regulation as critical to improving quality of online 
instruction. He focused on a leader's ability to stay current in the latest instructional design strategies for effectively 
promoting student learning. Design leadership competencies perceived by P6 include setting objectives and 
relevant, measurable assessments grounded in the learner’s context or situation; thus, a view toward “what the 
learner gets out of it.” Moreover, P6 was concerned with increasing student learning by affording analysis and 
problem-solving strategies and activities through interaction and inclusion of real-world tasks in the design 
affordances. 
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Abstract 

This investigation considers the effects of Interdependent and Independent group collaboration on the flow 
of information elicited as concept maps before (Premap), during (Group map), and after (Postmap) collaboration, 
and on the source, extent, and quality of Postmaps. Participants were undergraduate students (N = 37) randomly 
assigned by group (tetrads and a few triads) to the Independent condition who read and ‘mapped' an entire textbook 
chapter as homework or to the Interdependent condition who read the whole chapter but mapped either the first half 
or second half. After completing their Premaps out of class, group members collaborated face-to-face in class to 
further learn the content by creating a group concept map on a large sheet of newsprint. Then immediately, students 
individually drew their Postmap from memory. The Independent condition participants’ Postmaps had more 
idiosyncratic terms and were more like their Premaps relative to the Interdependent participants' Postmaps, and the 
Interdependent participants’ Postmaps were considerably more like their Group maps. Further, the Interdependent 
participants’ Postmaps were relatively more like the expert maps (62% vs. 52%) mainly due to a greater number of 
terms from the second half of the chapter, a possible primacy effect. Regarding knowledge convergence, the 
Interdependent Postmaps were relatively more like the Postmaps of others in their group (38% vs. 30% overlap). In 
this case. Interdependence resulted in less homework but nearly equivalent group maps and better and more similar 
Postmaps. 


Introduction 

People learn, formally and informally, and work in the company of others. In fact, collaborating in groups 
is a critical 21 st century job skill (Kozma, 2008) that has a natural fit with and a growing role in both face-to-face 
and online instruction. Group factors such as social cohesion and individual factors such as motivation, cognitive 
elaboration, and possibly developmental considerations (Slavin, 2010) all influence learning in groups. Much 
research and theory on group memory has originated in history, anthropology, and sociology, while group 
collaboration has been investigated by social psychologist and learning scientists with a strong orientation on 
describing and influencing group-level process variables such as roles and argumentation patterns. Recently there is 
a surge of interest by cognitive scientists and educational psychologist on the influence of group collaboration on 
individual cognitive activity such as retrieval, rehearsal, and forgetting (Rajaram & Pereira-Pasarin, 2010). 
Combining these approaches is consistent with Salomon and Perkins (1998) call to distinguish between the effects 
with and effects of collaborative learning. This present investigation is our beginning step to understand the flow of 
information pieces (as structural knowledge) in group collaboration as a way to look at how activity in the group 
influences individual cognition (note: our follow-up investigations already in progress employ video analysis of the 
group collaboration in an online setting to extend this line of research). 

Previous research has identified two counter-intuitive, negative, and perhaps even startling findings for 
group work. First, although the recall of a collaborative group is greater than that of any one member (Hinsz, 
Tindale, & Vollrath, 1997; Yuker, 1955), individuals working together in a group remember less than if they work 
individually and then pool their work. This is a well-established effect called collaborative inhibition . Second, 
Stasser and Titus’ (1985) seminal article on the hidden-profiles methods where group members are intentionally 
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provided only partial and different information that if combined provides a complete picture, found that group 
discussion tends to focus on information that members already share before the discussion and so members tend to 
perpetuate and even strengthen their initial knowledge at the expense of attaining a ‘fuller' understanding from the 
unique information brought by others. Frankly, these two well-established findings clearly suggest that group work 
would not be as effective as working individually and so these effects of working in groups must be qualified or 
moderated in order to recommend or design group work during instructional design. 

How does this play out? Almost certainly individuals enter into every group collaboration with both 
commonly shared information and also new unique information. A recent meta-analysis by Lu, Yuan, and McLeod 
(2012) summarized findings from 65 studies (101 Independent effects, 3,189 groups) that replicated previous 
findings that common information (i.e., information that is already held by several group members before the group 
collaboration) is mentioned more frequently than unique information (i.e., information held by one member of the 
group before the group collaboration) by two standard deviations more common information than unique 
information. 

Does collaborative inhibition occur in ordinary group collaborative learning? And do group members in 
collaborative learning predominately share common information relative to unique information? And what 
strategies, tactics, scaffolds, inducements, and incentives can instructional designers use to optimize collaborative 
learning for all members? Obviously one investigation cannot answer all of these questions. This investigation 
considers the flow of common and unique information before, during, and after group collaboration using a 
modified jigsaw method that resembles the hidden profiles approach. 

Jigsaw (Aronson, 1971) is a group approach that is popular and intuitively appealing but is time consuming 
and perhaps not always effective. In this approach, members are assigned different parts of a topic to study, and then 
they join a group discussion with others who have the same topic in order to more fully develop that topic. Then 
members rejoin their initial group where they become the group ‘expert' for that topic and members of the group 
‘coteach' each other the various topics. Jigsaw is thus a form of hidden profiles since different group members enter 
the group collaboration with different profiles or portions of the information. Jigsaw establishes interdependence 
between group members, since members must depend upon other members. Interdependent group tasks (i.e., from 
Social Interdependence Theory) typically engenders greater member interaction which can lead to more similar 
knowledge structure (Roseth, Saltarelli, & Glass, 2011, p. 812). 

Engleman and Hesse (2010) investigated interdependence in computer-supported collaborative learning. In 
their investigation, all the members of 40 triads (N = 120) initially individually studied partial information (i.e., 
hidden profiles) regarding how to control pests that are destroying pine trees in a forest, no one member within a 
triad had the full solution to the problem but the three members together would have enough information to solve the 
problem. The triads were randomly assigned to the Interdependent and Independent conditions. All triads met 
synchronously online to create a collaborative concept map using CmapTools software while talking via Skype, the 
Independent triads had access to all of the partial information given to the other two members of their triad while the 
Interdependent triads could not see the other members’ partial information. The Interdependent and Independent 
conditions interacted differently online and obtained different learning outcomes. The Interdependent triads on 
average spent 12.1 minutes creating a fully elaborated concept map before shifting focus to solving the pest 
problem, while the Independent triads spent only 3.9 minutes elaborating their map before focusing on solving the 
pest problem. Regarding learning, the Independent triads outscored the Interdependent triads on the solution to the 
fertilizer portion of the problem, although there was no difference between the two conditions on the pesticide 
portion of the problem. Analysis of the content and structure of the Group maps showed that the Interdependent 
triads’ maps were larger and more fully elaborated relative to the Independent triads’ maps. Specifically the 
Independent triads’ map had relatively fewer terms and links between terms, and those terms focused on the solution 
to the problem and left out information that was relevant but not directly problem related. Thus interdependence 
more fully drew out the partial information from each of the triad members, although ultimately those fuller map 
representations were less likely to prompt a solution to the pest problem. 

This findings for the Interdependent group (i.e., that they shared almost all of their unique information with 
the group) are notable since it is the opposite pattern of what would be expected regarding common and unique 
information based on Stasser and Titus’ (1985) that group discussion tends to focus on information that members 
already share before discussion. This may have occurred due to the use of concept maps as the group collaboration 
tool (Slithers, Vatrapu, Medina, Joseph, & Dwyer, 2007). In this present investigation, we desired to measure the 
flow of shared and unshared (unique) information before, during, and after group collaboration in order to explain or 
reconcile this pattern. 

In the present investigation, instructional treatments were developed to mimic a hidden profiles strategy 
(i.e., before group collaboration, learners focus on either the first half, last half, or all of a textbook chapter). The 
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learning strategies were sensitively negotiated with the course instructor to be learner appropriate in terms of 
content, level, task difficulty, and activity. In this we sought to bridge research and practice by establishing 
ecological validity in a real classroom while measuring information flow at an elemental level (Nathan & Alibali, 
2010). Concept maps (actually mind maps without relational link terms) were selected because of their utility as a 
note-taking device, as a semiotic tool for collaboration that can function as a “convergence artifact” (see Suthers et 
al., 2007), and as a measure that has concurrent validity with many measures of interest. Besides measuring the 
extent of knowledge in these pre, group, and post maps, we also measured the omnibus ‘structure’ of the maps as 
graph centrality (Clariana, Draper, & Land, 2011). 

Graph Centrality as a measure of Knowledge Structure 

Concept mapping consists of several distinct cognitive activities including recalling important terms (the 
extent of knowledge), sorting terms closer together or farther part (associational knowledge), linking highly related 
terms (relational knowledge), and labeling the links (propositional knowledge). This investigation considers the 
extent of knowledge as the concepts present in Premaps, Group maps, and Postmaps. In this report, we have not yet 
analyzed relational, associational, or propositional knowledge although we do provide graph centrality as one 
alternate measure of relational and associational aspects of knowledge structure (Clariana et ah, 2011). 

Specifically, Kinchin, Hay, and Adams (2000) have proposed that the overarching visual layout of a 
concept map, designated as spoke (star), chain (linear), and network (tree), provides a classification of cognitive 
structures that is a typology of thought (also Yin, Vanides, Ruiz-Primo, Ayala, & Shavelson, 2005). Hay and 
Kinchin (2006) further propose that spoke structures are indicative of a naive epistemology, chain structures are 
indicators of goal-orientation, and networks are indicators of expertise. Clariana et ah (201 1 ) quantified these four 
network graph layout forms using graph centrality as an omnibus numerical measure of concept map structure, with 
0.1 representing linear form, 0.4 a tree form, 0.6 representing a network form, and 1.0 representing a star form (see 


Figure 1). 
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Figure 1. Graph centrality calculated for four concept map forms from Clariana et ah (2011). 

Using graph centrality, Clariana et ah (2011) reported that Subaru employees in the field who learned 
together for several months in an online community of practice had a greater and highly similar graph centrality 
(Cgraph = -45, 66% overlap) for their written response to a problem-based case compared to participants who were not 
in the community of practice (C grap h = .25, 33% overlap). Thus the visual form of the participants’ knowledge 
structure in the community of practice strongly ‘converged’ (Weinberger, Stegmann, & Fischer, 2007). Their 
findings support the use of graph centrality as a measure of knowledge structure. 

Purpose 

The present investigation seeks to describe the influence of task interdependence on learners’ extent of 
knowledge as measured by Premaps, Group maps, and Postmaps and also the structure of their knowledge as 
measured by graph centrality. A follow up study underway now will replicate this approach online and will include 
video analysis of the group collaboration in order to describe the collaborative group process. 


22 


Method 


Participants . This investigation was conducted in a face-to-face section of CAS 250, Small Group 
Communication, at a large public Northeastern university. This is a required course for Communications majors. 
Eligibility criteria included course membership and voluntary participation with signed consent. The participants in 
this investigation were mostly sophomore undergraduate students between 19 and 22 years old. 60% were male. 
Students received course credit for completing the activities. Two students did not complete at least one of the 
activities and one chose not to participate, so the final sample is n = 37. The investigation was reviewed by the 
university’s institutional review board (IRB #38101). The 10 intact groups were randomly assigned to receive either 
the Independent condition or the Interdependent condition. 

Materials and Procedure . This investigation was conducted in weeks 9 and 10 of a 16 week semester. The 
students had worked in groups of four for most of the semester and so were familiar with each other and were 
accustomed to group work in class (ecological validity). 

During a class session in week 9, the principal investigator who is not the course instructor explained the 
investigation and led a group concept mapping practice session where the students read a 450-word text on a course 
related topic, “Social influence in groups” and then students worked in their existing groups of four to create a 
Group map of this content on large newsprint pads using yellow stickies and magic markers. 

Working alone as homework, participants in the Independent condition were asked to map all of Chapter 8 
(from the course textbook), while participants in the Interdependent condition were asked to map only the first or 
last half of the chapter. All were told to “do your map in order to support your team”. All were given this same list 
of 31 terms from the chapter that they could use for their homework Premaps. The handout for both the 
Interdependent and the Independent conditions stated, “Use any appropriate words in your concept map, but here are 
a few important words that we noticed in Chapter 8 that you could use too if you get stuck.” And it then listed these 
terms from Chapter 8, 18 terms for Reading 1 and 18 for Reading 2. Reading 1 (pp. 228-234) included: conflict, 
myths, disagreement, address or avoid, controversy, communication, idea deviance, group cohesiveness, influence, 
participation/ involved, consensus, decision making, goals, task conflicts ( substantive ), relational conflicts (affect), 
CMC trajectory, group think, group norms, and values. Reading 2 (pp. 235-244) included: management style, 
competitive, relational conflict, compromise, collaborative, integrative, avoidance, accommodation, appeasement, 
partial win-lose, win-lose, consensus, group norms, fairness, my needs, other’s needs, disagreement, influence, and 
group cohesiveness. Five of the terms were in both lists, including: disagreement, group cohesiveness, group norms, 
influence, and relational conflict. 

In the next regular class meeting, students met in their groups for about 45 minutes and created a Group 
map of the Chapter 8 content. When done, students handed in their Premaps and Group maps, and then individually 
completed a post assessment that consisted of creating from memory a concept map of the Chapter 8 content. The 
handout stated, “From memory draw a concept map of the chapter. Sometimes it helps to list the 4 or 5 most 
important words and place those first, then add to those as needed. Shoot for about 20 to 30 most important terms. 

Be sure to write your name on the page. You may use any appropriate words in your concept map, but here are a few 
important words that we handed out before that you can use if you get stuck.” (This is the same list handed out for 
the Premap above.) 


Results 

Five analyses are presented here: (1) the extent of knowledge measured as the number of concepts within 
groups across the Premap, Group map, and Postmaps; (2) the contribution of concept terms from members Premaps 
to their Group maps; (3) the contribution of concept terms from members Premaps and Group maps to their 
Postmap; (4) structural knowledge across groups measured as graph centrality describing Premaps, Group maps, and 
Postmaps; and (5) extent of knowledge convergence as measured by the average Postmap percent similarity of 
concept overlap of the members in each group. 

(1) Extent of Postmap knowledge flow from Premaps to Group maps to Postmaps 
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Figure 2. The average flow of Posttest concepts for each condition across Premaps, Group maps, and Postmaps, 
(note: Values shown in parentheses are comprised of all terms in the maps, not just Postmap terms). 

The average extent of Posttest knowledge for the Independent and Interdependent conditions in each 
concept map activity is shown in Figure 2. Many Posttest terms that had occurred in the Premaps (i.e., lnd. = 57%, 
Int. = 47%) were also present in the Group maps (i.e., Ind. = 42%, Int. = 39%). The Interdependent and Independent 
Group maps agreement with the expert were equivalent (i.e., 49% vs. 51%), although the Interdependent group 
Postmaps were a bit more like the expert maps (not significant) than were the Independent Postmaps (i.e., 62% vs. 
52%). 


(2) The contribution of concept terms from members Premaps to their Group maps 

To analyze Group map terms, a 2 x (3 x 5) repeated measures ANOVA was conducted with the between 
subjects factor Treatment (Interdependent and Independent) and the two within subjects factors Quality (Group map 
term occurs in Reading 1 expert map, Reading 2 expert map, or not in either expert map) and Premap (Group map 
term occurs in 0, 1, 2, 3, or 4 Premaps of that group). The interaction of Treatment and Premap was significant, 
F(4,32) = 3.956, MSe = 6.890, p = .010; this interaction is shown in Figure 3. As would be expected, since the 
Interdependent treatment members only mapped half of the chapter, most of the terms in their Group maps were 
only present in 1 or 2 of their Premaps (i.e., the same half of the chapter). Independent group members were 
instructed to map the entire chapter as homework, and most terms in their Group maps were present in 3 or 4 of their 
Premaps. It is important to note that the Treatment between subjects factor was not significant. Independent and 
Interdependent Group maps had equivalent quality and pretest source, although the Independent group maps were a 
little more like the expert maps (51%, refer back to Figure 2) than were the Interdependent group maps (49%). 
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Figure 3. The Group map significant interaction of Treatment and Premap. 


(3) The contribution of concept terms from members Premaps and Group maps to their Postmap 


A 2 x (3 x 2 x 2) repeated measures ANOVA was conducted on Postmap terms with the between subjects 
factor Treatment (Interdependent and Independent) and the three within subjects factors Quality (Postmap term 
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occurs in Reading 1 expert map, Reading 2 expert map, or not in either expert map). Premap (Postmap term occurs 
in that individual’s Premap or not), and Group map (Postmap term occurs in that individual’s Group map or not). 
Only treatment-related significant results are presented here due to space limitations. 

The interaction of Treatment and Quality was significant, F(2,70) = 4.524, MSe = 1.570, p = .014; this 
interaction is shown in the left panel of Figure 4. The Treatment and Quality interaction (left panel) shows that the 
two treatments had an equal number of important terms in their Postmaps from Reading 1 . But for Reading 2, those 
who mapped half of the chapter as homework ( the Interdependent treatment) had more of the important terms from 
Reading 2 in their Postmaps relative to those who mapped the entire chapter as homework (the Independent 
treatment). A follow-up analysis showed a significant interaction F(2,34) = 3.137, MSe = 3.68, p = .05, of Treatment 
(Interdependent first half of chapter. Interdependent last half of chapter, and Independent) and Quality (Reading 1 
expert map and Reading 2 expert map) and indicates that this posttest improvement in Reading 2 terms was due to 
those who mapped Reading 2 as homework, a primacy memory effect for reading and mapping (see the right panel 
of Figure 4). The good news is that this did not impair the posttest performance of those students who mapped only 
Reading 1 as homework, since their performance was equivalent to those who mapped the entire chapter as 
homework. And finally, the Independent groups’ posttests had more idiosyncratic terms (i.e., not in either expert 
map) compared to the Interdependent groups. 
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Figure 4. The Postmap significant interactions of Treatment and Quality (left) and follow-up analysis (right). 

Next, the interaction of Treatment and Group was significant, F(l,35) = 6.661, MSe = 2.151, p = .014; this 
interaction is shown in Figure 5. The figure shows that the Interdependent group members had more posttest terms 
that also occurred in their Group map relative to the Interdependent group, who strikingly had posttest terms that 
were NOT in their group’s map. This suggests that the Independent group members paid less attention to their 
group’s map, their Postmap was more dependent on their Premap and their own unique knowledge than on their 
Group map (refer back to Figure 2). 
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Figure 5. The Postmap significant interaction of Treatment and Group. 

(4) Structural knowledge (as graph centrality) 

Graph centrality values were calculated for all of the homework Premaps (37), Group maps (10), and all of 
the Postmaps (37). The Premap and Group maps were more linear in form (range . 16 to .20, see Table 1) while the 
Postmap forms were somewhat tree form (.27 and .31). We suggest that the smaller centrality values for the 
Premaps are likely due to the fact that the students were using these Premaps for “note” taking. People tend to take 
notes in a linear way, reading a paragraph and mapping it, reading the next paragraph and mapping it, and so on. The 
smaller values of the Group maps may also be a reflection of this note-taking effect, group members have their 
Premaps out during group mapping and so may be reflecting the structure of their Premaps onto their Group maps. 
Alternately, perhaps Group maps just tend to have this form of structure if members contribute larger or smaller 
“chains” of information in chunks to the group map. 

The Postmaps obtained a larger centrality value (i.e., .27 and .31) than the Premaps and Group maps 
(significant). Recall that participants did not have any notes or maps when drawing their Postmap, Postmaps were 
drawn from memory. Perhaps maps drawn from memory tend to obtain this level of centrality innately or this 
specific content may interconnect more when drawn from memory (i.e., larger centrality). 

Table 1. Average graph centrality (standard deviation shown in parenthesis) and correlations for each Condition and 
Task. 


Average Graph Centrality 


Interdependent 

Preman 

.20 

Group 

map 

.17 

Postman 

.27 

(n = 18, 5 groups) 

(TO) 

(.12) 

(.17) 

Independent 

.18 

.16 

.31 

(n = 19, 5 groups) 

(.13) 

(.09) 

(-20) 


(5) Extent of Knowledge Convergence 

The Postmap percent similarity for members within each group was calculated as the ratio of terms in 
common between two Postmaps divided by the average total terms in the two Postmaps (see Table 2). Then these 
values were averaged to obtain the group’s Postmap average percent similarity (not the same as graph similarity). 
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Table 2. Postmap average percent similarity for the members in each group. 



Interdependent 


Independent 

Group 2 

28% 

Group 1 

22% 

Group 3 

49% 

Group 5 

31% 

Group 4 

38% 

Group 6 

36% 

Group 8 

33% 

Group 7 

30% 

Group 9 

41% 

Group 10 

31% 

M 

= 38 % 


30 % 

SD 

= 0.08 


0.05 


The Interdependent condition average percent similarity is 38% and for the Independent, the average 
percent similarity is 30%, an effect size of 1.03 (t-test p =.11). Although this difference is not significant, the trend is 
in the anticipated direction that Interdependent groups’ members' knowledge extent converges more relative to the 
Independent group as reflected in more similar post-lesson concept maps. 

Discussion 

This investigation considered information flow in Interdependent and Independent task collaborative group 
work measured using Premaps, Group maps, and Postmaps. For the Premaps, not surprisingly map sizes were on 
average about half as large for the Interdependent condition ( 19.0 terms) compared to the Independent condition 
(36.3 terms). This is consistent with mapping half of the chapter versus mapping the whole chapter. Group maps 
were approximately equivalent in terms of quality and size. Interdependent group agreement with the expert was 
49% for 32.5 terms compared to 51% for 28.1 terms for the Independent group maps. But flow of terms from 
Premap to Group maps varied by condition, the Interdependent Group map terms had only occurred in 1 or 2 
members' Premaps while the Independent Group map terms had mainly occurred in 3 or 4 members’ Premaps. 
Simply stated, the members brought forward to the group terms form that part of the text that they had previously 
studied (read and mapped). Regarding Postmaps, first the Group maps were considerably larger than the Postmaps. 
Next average Postmaps for each condition were approximately equivalent in terms of size but not in terms of 
quality, the Interdependent agreement with the expert was 62% for 16.4 terms compares to 52% for 17.1 terms for 
the Independent group maps. The improvement in quality for the Interdependent condition appears to be due to a 
relative increase in Postmap terms from the last half of the chapter. This suggests a primacy effect for reading and 
mapping where what is covered first is better recalled. The quality of the Independent maps was also reduced due to 
the increase in idiosyncratic or unique Postmap terms that were not in the expert’s maps. 

These findings indicate that the Interdependent group members must have learned the other half of the 
chapter that they had not studied through the group collaboration and what they learned in the group tended to be of 
higher quality. Specifically, 21% of the terms in the Interdependent groups Postmaps came exclusively from their 
Group maps while only 4% of the Independent group members’ terms came exclusively from their Group maps. 

Another striking difference is the large percent of Posttest terms that were in neither the individual’s 
Premap or their Group map, 39% in the Independent condition and 32% in the Interdependent condition. This 
suggests that individuals have idiosyncratic information that is not ‘active’ enough to include in their Premap or to 
share in the Group map but that was worthy of including in their Postmap. A parsimonious explanation is that this 
low activity unique information in all individuals is slightly suppressed at Postmap by Interdependent group 
collaboration (rather than that it is enhanced in Independent group collaboration). The bigger question is why do 
members share some of their information but not all of it? This seems important, so it needs further analysis to 
determine what kind of information this is. 

This investigation has a number of limitations. First, this investigation used open-ended concept maps 
where participants may use any terms they. Open-ended concept maps are the gold standard but are notoriously 
difficult to analyze due to factors that introduce analysis error such as having to manually standardize concept terms 
across all maps to control for synonyms and metonyms (a big and daunting task), and analysis comparison problems 
due to the highly varied map sizes (in this study map size ranged from 6 to 23 terms in the Interdependent Postmaps 
and from 10 to 29 in the Independent Postmaps) that effect math calculations (e.g., imagine the complexity of 
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analyzing multiple-choice posttests where participants answered very different numbers of question: 5 out of 5 
would be 100% while 7 out of 10 would be 70%, so who did better? 5/5 or 7/10?). 

The expert’ s maps of readings 1 and 2 used as the benchmark to evaluate the quality of all maps are also a 
limitation. How good are the expert maps? Different expert maps produce different quality values for every student 
map. So these quality measures reported here must be interpreted with caution and be viewed only as general 
estimates of map quality. 

Another substantive limitation is the use of only concept maps, other measures of post-group knowledge 
are needed in order to describe the learning effects of these strategies and the possible relationship between the map 
measures and these other measures. The graph centrality measures used here are very speculative, more research is 
needed to establish this approach as a valid and operational measure of knowledge structure. 

Also, these findings for information flow in this face-to-face group collaboration may not generalize to 
online collaboration, which is our main research interest. 

Contribution to practice . These tentative findings suggest that this modified jigsaw strategy does lead to 
quality knowledge sharing in group collaboration and requires less homework overhead (e.g., map half the chapter 
for homework rather than the whole chapter). Thus this co-teaching group collaboration approach may be very 
appropriate for online courses that cover required textbook content. 
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Abstract 

The inclusion of fully online courses in higher education degree programs is growing. Although most students 
accept these online courses, other students express strong preferences for traditional F2F environments. These latter 
students are averse to online learning. When they are forced to take online courses to complete a degree, the averse 
students express high dissatisfaction with their courses, especially concerning learning and retention. This study 
sought to identify what web averse students mean when they are dissatisfied with learning and retention. It also 
sought feedback on course elements web averse students found satisfactory. 

Dragging the Y Generation Into E-Learning: A Study of Engaging Averse Online Learners 

Growth in technology and social media has occurred concurrently with a surge in online university courses 
(Allen & Seaman, 2008; Beqiri, Chase & Bishka, 2010) and degree programs. Gen Y college students (i.e., persons 
born in the 1980’s and 1990’s) are high users of social media, smart phones, and other technologies. Contrary to 
their strong connections to technology, some Gen Y students strongly prefer face-to-face (F2F) college courses and 
select academic programs specifically for the traditional F2F delivery format. Learners who openly prefer F2F 
courses and lack interest in enrolling in online course are considered web averse. Although web averse students 
prefer to avoid online learning, some F2F degree programs require courses which are offered only in an online 
format. This situation is likely to continue and grow as colleges and universities continue to expand their online 
course offerings (Allen & Seaman, 201 1). “Thirty-one percent of all higher education students now take at least one 
course online” (Allen & Seaman, 2011, p. 4). In community colleges, this rate nearly doubles. Gonzalez (201 1) 
reports that six out of ten community college students have taken a class online. 

The Department of Psychiatric Rehabilitation and Counseling Professions (Department), at the University 
of Medicine and Dentistry of New Jersey (UMDNJ), School of Health Related Professions has been providing 
online courses for almost 15 years. In the undergraduate programs, students take most of their courses as a cohort in 
traditional, F2F classrooms. The students become familiar with faculty and develop strong relationships with peers 
and instructors. These students are also required to complete two fully online courses. 

A review of prior years’ student satisfaction surveys and existing alumni focus group data from the 
Department revealed that 1) many students made statements indicating they were web averse, and 2) these web 
averse students were dissatisfied with the amount they learned and retained in their online courses. These 
satisfaction surveys indicated that students felt online courses required more reading in texts, journals, online 
postings, and websites, and provided less contact with the instructor and classmates. Similar to problems identified 
in the literature, the UMDNJ students reiterated that the online courses took far more study time than their F2F 
classes (Fey, 2006; Fey, Emery & Flora, 2009). Ultimately, they reported significant dissatisfaction with the amount 
they learned and were not confident they would retain what they learned. 

Study Background 

This study was conducted during the fall 2009 semester. It focused on one required undergraduate course. 
Community Resource Management, which examines the Community Support System for people who have serious 
and persistent mental illnesses. This course is fully online for two principle reasons: 1) The online environment 
allows students the opportunity to explore real community programs, services, and practices, while also learning the 
information in the same context in which they will later use it (Smith & Ragan, 2005; Lindsey & Berger, 2009). 

This contextual exploration maximizes students’ ability to apply their learning in the future (Merrill, 2009, p. 2) 
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Students learn to use the Internet to find the most up-to-date information available about current resources in their 
own community. Searching out current resources is a critical skill for practitioners in mental health (Pratt, Gill, 
Barrett & Roberts, 2007). Additionally, the course is offered online because it allows students flexibility during 
their practicum semester when many students are away from campus. 

Prior studies have examined characteristics of learners (Beqiri et al., 2010; Changchit & Klaus, 2008; Eom, 
Wen & Ashill, 2006) readiness for online learning (Chu & Tsai, 2009), time management (Roper, 2007; Terry & 
Doolittle, 2008), and engagement of learners in online courses (Carle, Jaffee, Vaughan, & Elder, 2009; Huang, 

2010; Martin & Dowson, 2009). However, there has been little research on web averse learners who must take 
online courses to obtain their degree. This project had two primary strategic aims: 1) Identify what students mean 
when they state they are not learning enough or retaining information in their web courses. 2) Identify what web 
averse students find helpful in online courses. 

Conceptual Framework 

“The fundamental idea underlying engagement theory is that students must be meaningfully engaged in 
learning activities through interaction with others and worthwhile tasks” (Kearsley & Shneiderman, 1999, p. 1). 
Engaging web averse students through meaningful interactive components may be helpful for improving their 
learning experience. The literature shows that inclusion of interactive components contributes to learner 
engagement (Betts, 2009a; Betts, 2009b; Chickering & Gamson, 1987; Council for Regional Accrediting 
Commissions, 2000; Silverthorn, 2006). For online courses, interactive components are design elements that require 
a response from the learner. These components may include discussion forums, live chats, interactive learning 
objects, or other elements. The inclusion of interactive components within the course helps learners remain 
involved in the learning process (Wang, 2009) and may contribute to increased satisfaction with their learning 
experiences. 

When they are required to take courses offered only in online formats, web averse students in the 
Department show poor participation rates in their online courses (P. Basto, personal communication, September 3, 
2009). They often fail to access interactive components and have low participation in learner-learner and instructor- 
learner interactions, e.g., discussion boards. Ultimately, these students report significant dissatisfaction with their 
online learning experience. They voice complaints formally in evaluations and informally in F2F conversations 
with classmates and other faculty. Some students withdraw because they are not able to commit the necessary time 
for the courses (Fey et al., 2009). Engaging web averse learners may be challenging, but has potential for helping 
them learn and be more satisfied with their online learning experience. 

Methods 

The online course used for this study had 17 enrolled students. These students came from AS (8), BS (8), 
and MS (1) programs in psychiatric rehabilitation within one department. A few students had previously taken 
online courses, but most had no experience with an online classroom. IRB approval was obtained through the 
UMDNJ. A F2F orientation, led by the course instructor, was held on campus during the first week of the course. 

Several methods were used to gather information about the students’ experiences in the online course. 
These methods included pre-course and post-course research surveys, and focus groups at midterm, and post-course. 
Additionally, data was gathered from course polls and a 16-item internal course survey routinely used in courses for 
formative evaluation purposes. All research surveys and internal course surveys were anonymous and voluntary. A 
35 item student research survey was administered via online survey software (Survey Gold) after the F2F course 
orientation. This survey was composed of Lickert-type questions, some of which requested further comment in an 
open-ended format. The survey items were developed from prior course surveys used in the Department, as well as 
surveys from other institutions. Several items in the internal course survey overlapped in content with the research 
surveys and focus groups. 

Two focus groups were also conducted. The first focus group occurred at the semester’s mid-term. While 
all course students were invited to participate in the focus group, only five attended. At the end of the semester, six 
students participated in a similarly formatted post-course focus group. Questions for both focus groups sought 
information on student perceptions of learning and retention. A light meal and $25.00 gift card were offered as 
incentives for participation in each focus group. 

In addition to the anonymous research surveys and focus groups, information for this project was obtained 
through the Angel LMS system via anonymous internal course surveys, internal student tracking, and internal 
student polling. Ten polling questions were placed individually or in pairs on the course homepage throughout the 
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semester as a formative evaluation process for the course. These polls invited students to offer information about 
their experience and make suggestions for the course. The polls focused specifically on engagement techniques and 
student learning and retention. 


Results and Discussion 

Demographic data was collected on the pre-course research survey, for which there were only seven 
responses. No individual student identifiers were collected on any data sources. Response rates to the research 
surveys, focus groups, and internal program evaluation sources varied widely, with the greatest number of responses 
coming from the internal course surveys and polls. 

Course Experience 

Table 1 shows response comparisons for students’ overall experience with the online course. 

Table 1. Response Comparisons for Students’ Overall Experience 


Question from survey 

n 

Highly 

Dissatisfied 

Dissatisfied 

Neutral 

Satisfied 

Highly 

Satisfied 

Post-course Research survey: 

How would you describe your 
experience with this online course? 

8 

0 

0 

4 

4 

0 

Post-course Internal Survey: How 
would you describe your experience 
with this online course? 

12 

0 

2 

6 

3 

1 

Post-course Internal survey: 

This online course has been an 
effective learning experience for me. 

12 

1 

0 

4 

6 

1 


Responses to questions regarding students’ experience with the course varied considerably. Half of 
students responding to the post-course research survey indicated they were satisfied with their course experience, 
with the other half selecting neutral. In contrast, only about a third of respondents on the internal post-course survey 
were satisfied and half were neutral. Just over half of student respondents in the post-course internal survey stated 
that the course was an effective learning experience. One third of students identified a neutral response to the same 
question. The number of students expressing satisfaction at the end of the course was higher than expected for this 
group of students. 

Responses regarding satisfaction with the amount learned and retention were similarly surprising. Table 2 
shows the responses to questions regarding satisfaction with amount learned and retention. 

Table 2. Satisfaction with Learning and Retention 


Question from survey 


Strongly Disagree Neutral Agree Strongly 

n Disagree Agree 


Satisfaction with Learning 

Post-course Research survey: 

I learned the same amount of course content in 8 2 2 0 4 0 

this Web-based course as I would have if it 
was offered in the traditional classroom 
setting. 

Post-course Research survey: 

How satisfied are you with the amount of 8 0 1 14 2 

learning you have GAINED from this course? 
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Post-course Internal Survey: 


How satisfied are you with the amount of 
learning you have GAINED from this course 
as compared to other courses in the 
Psychiatric Rehabilitation program? 

12 

0 

0 

4 

5 

3 

Satisfaction with Retention 

Post-course Research survey: 

I retained the same amount of course content 
in this Web-based course as I would if it was 
offered in the traditional classroom setting. 

8 

2 

1 

2 

1 

2 

Post-course Internal Survey: 

How satisfied are you with the amount 
of learning you have RETAINED from this 
course as compared to other classes in the 
Psychiatric Rehabilitation program? 

12 

0 

0 

4 

6 

2 

Post-course Research survey: 

How satisfied are you with the amount of 
learning you have RETAINED from this 
course? 

8 

0 

0 

4 

4 

0 


On the post-course research survey, there was an equal amount of responses, between satisfied and dissatisfied, 
when asked about the amount students had learned as compared to a F2F classroom. However, responses to similar 
questions about the amount of learning gained consistently showed high levels of satisfaction. Questions about 
students’ perceptions of retention on the post-course research survey also resulted in an equal amount of responses 
between satisfied and dissatisfied. However, a quarter of respondents reported they were neutral. In contrast, the 
post-course internal survey showed that two-thirds of students were satisfied and one third were neutral about the 
amount of learning they retained. Half of respondents to the post-course research survey were satisfied with the 
amount of learning retained, while the other half responded neutral. 

Although student satisfaction was higher than expected for learning and retention, student preferences for 
F2F classes remained about the same. These preferences were reflected in comments such as: “Not that it wasn't 
successful, it is simply that I prefer in-person classes.” “I believe traditional classroom setting is the best way to 
learn.” “I prefer personal interaction and take more away from it.” “. . .my learning style is more visual and I like 
being taught rather than having to take on all responsibility myself.” 

Students 


A majority of students began the semester with negative views of online classes. By the close of the 
semester, the students had some positive experiences in the online environment but continued to state clear 
preferences for F2F courses. Ultimately, these students were satisfied with their learning and retention in the online 
course. Many reported that the course was an effective learning experience, and they found the flexibility and 24/7 
availability of online learning attractive. However, they were unhappy about the amount of reading and study time 
required, and stated that they had to work somewhat harder in this online course than in traditional F2F courses. 

This satisfaction, despite the struggle, may be related to their transition to increased responsibility for their learning 
process, or to other factors such as time management, course design elements, or sense of distance from the 
instructor. 

Moore and Kearsley (2005) state, “Students frequently do not understand that they must take a large degree 
of responsibility for their learning in a distance education course. . . This kind of misunderstanding leads to students 
falling behind and becoming dissatisfied ( p. 178). ” Eom et al., (2006) also place responsibility for learning and 
satisfaction heavily on the student, stating that self-motivation contributes to successful students, while less 
successful students do not have the same self-motivation. Students must be active learners to be successful in online 
learning: “learners must self-manage the learning process (p.216).” 
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The ability to decide when to study, determine what is most important in the reading or activities, and self- 
guide other learning tasks is critical to success in online learning (Moore & Kearsley, 2005). Students in this study 
were divided regarding their self-direction in learning — defined in this study as scheduling their own time to 
complete readings or do assignments, and planning study time. The students reported feeling overwhelmed and had 
trouble establishing and maintaining study routines. They struggled with finding time for study, deciding how to 
proceed with assignments, and tracking their progress on weekly readings and assignment postings. It is likely that 
at least part of this difficulty was related to the novelty of online learning to these students. Trying to learn new 
study skills and habits at the same time they were trying to absorb the course content and complete learning 
activities may have frustrated these web averse students (Driscoll, 2005; Mayer, 2005; Morrison & Anglin, 2005). 
One student attributed his/her anxiety about taking an online course to “time management, getting everything read 
thoroughly and being able to complete assignments on time.” In the post-course survey, another student concern 
was, “Adapting to learning all of the material by myself.” 

In the mid-term focus group and post-course survey, the strongest theme across all data sources was the 
challenge of time management. This was not surprising, as other studies have also identified this issue (Roper, 

2007; Terry & Doolittle, 2008). Some students stated that they were not able to manage their time effectively in 
order to meet course requirements. Students reported having difficulty setting time aside to go online, read required 
materials, browse required websites, and participate in online discussions. In the post-course survey, one student 
stated, “Having the ability to schedule my own time on the computer was difficult.” In response to a post-course 
survey question about the success experienced in the course, a student stated, “I am not disciplined enough to 
continuously follow the lesson plan.” 

Some students became more comfortable with asynchronous online learning as the semester progressed. A 
student commented, “Once I got use to the format, the online class was ok.” Another stated, “I'm not particularly a 
fan of online courses, however, this may be because I haven't taken many of them and it may be just a matter of 
acclimating myself to this format change.” One student responding to the post-course research survey commented, 
“The weekly quizzes caused me to really stay on top of my work and my reading and to study hard. The discussion 
question really made me analyze the material that I read, learn and studied for the quizzes.” One respondent of the 
internal post-course survey summed up the experience, “For all that I have posted here, I am really satisfied not only 
with how much I learned, but also become it was a great on-line class with much class participation and instructor's 
incentives and participation! Thanks [instructor]!” 

Other students continued to prefer traditional F2F learning environments. In the post-course survey, one 
student stated, “I learned a lot about community resources but I rather to be [sic] in a classroom, that is the way 
where [I] learn more.” Another wrote, “I will not take another online course. This has nothing to do with the 
professor. I just do not feel like I learn through online.” 

Course and Design Elements 

Pedagogy in online classrooms is different than in F2F classrooms. One of the pedagogic changes is the 
shifting of roles for both instructor and learner. Instructors must change their techniques from the objectivism of 
“housing all necessary knowledge” (Eom et al., 2006, p. 219) to an empowering, collaborative facilitation of the 
current online constructivist environment. Realigning teaching strategies from a traditional direct instruction 
approach to a constructivist approach has been an iterative process for the Community Resource Management 
course used in this study. Evolving over time, the course has been taught and updated by several professors. 

Faculty at the university develop their own online courses and. at the time of this study, none had training in 
instructional design. As faculty identified new resources for the course, the quantity of course materials 
accumulated over time. 

Although reading assignments may have been longer than similar courses, students also sometimes grossly 
overestimated the amount of required reading. According to one student in the post-course focus group. 

So much reading, so I had little retention because it was so much. For example, module 14 in 
particular there was so much to read in two weeks it was impossible to retain the information. 

Reading was overwhelming, sometimes hundreds of pages. I could not read it all. Not enough time 
in the day to read it all. There was a lot of reading and was way over the three hours for class. 

(anonymous student response) 

The actual reading assignment mentioned by the student was the largest in the course. It was 80 pages assigned over 
a two week period. Over-estimation of reading assignments was more prevalent when the assigned reading was a 


34 



small part of a larger document, e.g., a chapter within a lengthy report or an executive summary of a longer 
document. When students clicked on documents without reading the instructions (or consulting the Learning 
Guide), they mistakenly thought they had to read the entire document. Although the actual volume of reading was 
not substantially more than that of other courses, students developed an initial (false) impression that they had 
excessive reading assignments. Students who printed their reading assignments were notably more vulnerable to 
false impressions about the volume of reading assigned. This was likely because they would click and print an 
entire document without taking the time to read the instructions. If the instructions were read, students would have 
been informed that only one section of the document needed to be read in full. 

The presence of numerous, easily accessible, supplementary optional resources may also have contributed 
to students feeling overwhelmed. For example, most weekly lessons included links for optional articles and Internet 
resources that were labeled as Additional Resources or Optional Resources. In F2F classrooms, additional resources 
are often limited to lists on the course syllabus and thus may not be as conspicuous and accessible to students. 
Students for whom online learning was novel, and who may have already felt overwhelmed by the online course, 
may have perceived optional materials as additional requirements for the course or been distracted by the resources, 
further contributing to student perceptions of excessive reading. 

Clearly defining the purpose for additional or optional resources in the course’s first week could help deter 
students from being overwhelmed. Weeding out and prioritizing those resources that offer the greatest benefit 
would also reduce the potential for students’ misconceptions about the quantity of reading materials. Changchit and 
Klaus (2008) found that perceived difficulty and usefulness affected student preferences regarding F2F and online 
formats. If extensive resource lists increase student perceptions of difficulty, then perhaps limiting the quantity of 
materials — even materials labeled as optional — could be helpful. 

Similarly, access to assignment instructions must be easy and very obvious for novice and averse online 
students. In the F2F classroom, students are accustomed to the instructor explaining assignments during class. The 
assignment instructions in the study’s course were available in detailed step-by-step written directions (Word 
document) and supplemented with a narrated PowerPoint. Students in this study easily became overwhelmed when 
reading instructions for the course project. Some contacted the instructor for clarifications, while others relied on 
classmates to understand what the assignment required. Although detailed instructions are generally considered 
beneficial (Rothwell & Kazanas, 2008; Smith & Ragan, 2005) these students may have felt inundated by the number 
of instruction links. Combining all instructions and forms into one document and emphasizing the importance of 
thorough reading of that document could reduce the students’ sense of reading burden. During the week that the 
project begins, the instructor might also include an emailed audio welcome message to introduce the project and key 
components. 

In the post-course internal survey, which had the highest number of responses for any data source, students 
were asked to identify elements of the course, which were satisfactory and dissatisfactory. Respondents were 
offered two separate lists, one for items they found satisfactory and one for items they found dissatisfactory. The 
same 1 1 items appeared on both lists, in the same order. Table 3 shows the items and student responses. 
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Table 3 Satisfaction with Course Design Elements 


Design Element 

Satisfied 

Dissatisfied 

Textbooks 

3 

5 

Online module materials 

10 

1 

Discussion forums 

7 

3 

Live chat session 

1 

7 

In-person orientation session 

2 

2 

Instructor feedback on assignments 

1 

4 

Flexibility of instructor 

4 

0 

Availability of instructor via email and phone 

5 

2 

Technology and its support 

1 

4 

None of the above 

0 

1 

Phone meeting(s) with instructor 

4 

0 


Note. n=12 


Of the 12 students responding, 10 selected the online module materials, which consisted of a conversationally 
written lesson on the week's topic. The lessons heavily supplemented the textbook, which was later discontinued 
due to poor student reception. Seven of 12 respondents identified discussion forums as helpful for learning. These 
forums usually required analysis or application of the lesson content, and students had to make three posts for each 
question. The discussion forums were the most social aspect of the course, so it is not surprising that students who 
preferred F2F interactions enjoyed this teaching strategy. Burnett, Bonnici, Miksa, and Kim (2007) also support this 
perspective, stating “Group discussion online contributes to community building by bringing people together 
through ideas, concepts or theories” (p. 24). 

Each week the instructor placed reminders of important events or content on the course homepage. Eight 
of 12 students responding to the post-course internal survey identified these reminders as effective for their learning. 
A weekly learning guide (Smith, 2008) provided students with a required and recommended reading list, required 
activities, learning objectives, a self-assessment, and information on upcoming graded assessments. Although some 
students reported that they did not access the learning guides, those who did use them reportedthey were useful for 
organizing study time and staying current with assignments. One respondent stated, “The learning guides and the 
module guides were extraordinarly helpful and really helped clarify and tie together our reading [sic] material.” 

Some students suggested the learning guides include a checklist of tasks so students could track activity/assignment 
completion. Placing the content of the learning guide where students view it automatically (e.g., in the weekly 
lesson homepage) might increase its use while simultaneously reducing students' perception of numerous documents 
to read. 

Weekly quizzes received the highest marks on the internal post-course survey for helping students learn 
more effectively, receiving endorsement from 10 of 12 students. A brief, multiple-choice quiz was administered at 
the end of most weeks in leiu of a midterm exam. The immediate feedback from quizzes was helpful to the students 
as well as the instructor. Students were able to ask questions to clarify course content and the instructor was able to 
reinforce or repeat information to improve learning. Frequent assessments also provided students regular 
reinforcement for remaining current with reading assignments and other learning activities. 

Several times during the course, a live chat session was offered as an opportunity to ask the instructor 
questions or review for an exam. These sessions moved quickly. For example, a live chat session was used as a 
final exam review. The session lasted approximately 90 minutes and had 418 messages. This pace was far too fast 
for many students who had difficulty keeping pace reading posts and responding. It was not surprising that live chat 
sessions received the highest dissatisfactory rating in the post-course internal survey. 

The newly implemented Angel LMS was unfamiliar to all students, resulting in some minor technology 
issues and temporary frustrations. Of note, however, were accessibility issues for students who had visual 
impairments. “Due to my impairmrnt [sic], the set up of angle [Angel] [sic] causes me to get headaches,” according 
to a post-course research survey comment. Although the course employed universal design (Stone, Jarrett, 
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Woodroffe & Minocha, 2005) for instructor-controlled items, some students with visual impairments had difficulty 
using their usual screen reading programs or other adaptive devices within the course. These issues were referred to 
a Departmental task force for further investigation. 

Instructor 

One strength consistently noted by students was the instructor’s flexibility. Most students in the study had 
jobs and many had family responsibilities. Additionally, students enrolled in the study’s course were concurrently 
enrolled in a 240-hour clinical practicum. Thus, the instructor’s flexibility was key to maintaining contact with 
students. Weekly reminder emails from the instructor and pre-arranged midterm phone meetings were also noted as 
helpful for most students and contributed to their satisfaction with the course. 

Students also consistently reported that the instructor had good availability: “The instructor was very 
available. I had some problems this semester and the instructor was very helpful. The instructor was very 
understanding if I needed more time.” However, some students were still hesitant to make contact. For example, 
one student in the midterm focus group responded, “I’ve been reluctant to contact the instructor especially about 
clarifying assignments and discussions. I feel more comfortable asking someone in person instead of through e- 
mail.” 

One common issue in asynchronous online courses is the delay students experience in receiving feedback 
(Moore & Kearsley, 2005), particularly in discussion forums. Students who are more accustomed to F2F courses 
are used to receiving instantaneous feedback from their instructor and classmates. At the beginning of the course, 
the instructor established periodic phone meetings individually with each student to decrease issues of transactional 
distance (Moore & Kearsley, 2005). Many students reported that phone contact with the instructor was helpful in 
meeting their need for personal and immediate feedback. One student responded to a course poll, “Having the 
midterm phone meeting went very well. You addressed what I needed to improve so I'm to able to bring [sic] grade 
up... Thanks for taking the time to clearly explained it.” 

However, phone meetings and availability by cell phone did not always meet students’ needs for immediate 
interactions. As one post-course focus group participant stated, “I lost the intended trigger response that would get 
me to talk or understand material by being online instead of in person. I missed the verbal communication and 
contact that occurs in a classroom setting.” Eom et al. (2006) states, there is a “clear relationship between instructor 
feedback and student satisfaction and perceived outcomes” (p. 229). Changchit and Klaus (2008) support this with 
their finding that lower retention rates for online courses can “be attributed to reasons such as lack of personal 
interaction, inexperienced faculty, students unaware of the expectations, and students with multiple obligations” 
(2008, p. 35). 

Students and instructors have very different perspectives on acceptable timing for instructor feedback. Ni 
and Aust (2008) stated that verbal immediacy (i.e., sense of closeness) with the instructor increases the sense of 
community with the students in online discussion forums. Burnett et al., (2007) also supported this position. Some 
students, however, have expectations that they will receive feedback within minutes or a couple of hours whenever 
they contribute a discussion posting or submit an assignment. This is unrealistic because, unlike the course website, 
instructors cannot be available at all times. Rochester and Pradel (2008) noted similar concerns, “students expected 
immediate responses to emails since a timeframe for the course master feedback was not addressed on the syllabus” 
(p. 6). This study’s post-course surveys further revealed that students were dissatisfied with amount of teacher 
interaction and delay in receiving feedback, even though it was usually delivered within 48 hours. 

Limitations of the Study 

Student participation in the focus groups and surveys was lower than expected. Incentives were provided 
for participation in focus groups, however interest remained low, with less than 50% attending. Personal identifiers 
were not collected in the focus groups, so it is not known whether these students were the same, different, or 
overlapping for the two groups. The representativeness of these groups to the whole is also questionable. The 
number of students who completed the pre-course and post-course research surveys was also much lower than 
anticipated. For example, less than half of the 17 enrolled students participated in the pre-course research survey 
when demographic information was collected. This information could have been collected at other points in the 
study to provide a better view of the enrolled students’ characteristics. Some of these limitations were mitigated by 
the use of other data, such as course polls and anonymous in-course (internal) surveys, which were built into the 
course by the instructor. Participation in these was much higher, increasing the representativeness of study findings. 
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Conclusion 


Students who are averse to online learning offer a unique perspective to online learning. Surprisingly, 
students in this study were generally more satisfied than dissatisfied with their learning and retention. Although 
participants expressed satisfaction, they continued to prefer the F2F learning environment. Students in this study 
had a particularly difficult time adjusting to the greater volume of reading necessary for online learning. They also 
struggled with reading, interpreting, and following instructions for weekly assignments and course projects. 
Communication delays in the asynchronous environment also frustrated some students who expected immediate 
responses to discussion posts and emails. 

However, these findings do not explain what students mean when they say they are satisfied with their 
learning. For example, it is not known whether students who say they are satisfied with their learning mean that 
they fully understand the concepts, can apply the skills in real world situations, or can integrate their new knowledge 
with existing knowledge. Authentic assessment (Berg, 2006) can provide an objective perspective on this issue; 
however, it may not be consistent with the student view of learning. Students who get A’s do not necessarily feel 
satisfied with what they learned. Further research should pursue clarification of what students mean when they 
express satisfaction with learning. 

Even well-designed and delivered online courses may pose challenges for web averse students. These 
students easily become overwhelmed and their frustrations can quickly spread to other students, affecting the 
course’s community. Achieving success in the online learning environment requires students to adjust to the 
increase in responsibility for their learning. In addition, students must adapt their learning styles to the new 
instructional activities involved in online learning. Instructional designers and course faculty must carefully 
consider every element included in courses in which averse online learners may be enrolled. Extraneous material, 
even when it has instructional value, can contribute to some students feeling overwhelmed. When sections of 
documents serve as reading assignments, the assigned pages should be bookmarked or extracted to eliminate 
confusion. Key instructions and essential course elements must be extremely brief and conspicuous. Project 
instructions must be succinct, yet detailed. Personalized and encouraging emails, discussion replies, and course 
messages can help reduce anxiety for web averse students. Personal phone contact can also be beneficial for 
establishing rapport with unfamiliar students and discussing their concerns about taking an online course. These 
recommendations are consistent with best practices for design and teaching, and they attend to the heart of concerns 
expressed by averse online learners. 


Acknowledgement: Peter Basto was a co-investigator in this study and contributed to this paper. 
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A Mixed Method Case Study: How Principals Cultivate Technology 
Integration and use Social Networks for Professional Growth 

Barb Brown and Michele Jacobsen, University of Calgary 

Abstract 

In this case study, the innovative leadership practices and actions of principals involved in making school- 
wide improvements with technology in teaching and learning were examined. Ways in which principals 
utilize social and technological networks to support educational reform as a process of teaching and 
learning improvements were explored. Interview and survey data from principals and teachers in three 
school districts engaged in innovative reform involving educational technology over the duration of one 
school year informs the primary research question, “How do principals cultivate teaching and learning 
improvements integrating technology that meet the needs of today’s learners?” 

Overview of Theoretical Framework 

Digital media and information technology sponsors major cultural changes in education that prompt the 
need for shifting mindsets in educational theory, pedagogy and leadership. Research on technology integration 
efforts in schools have yet to fulfill the expectations and optimism for instructional improvements and positive 
impacts on student learning (Bransford. Brown & Cocking, 2000; Cuban, 2001; Dede, 2007 a, 2007b; Sawyer, 2006; 
Schacter, 1999). School principals are key instructional leaders for school reform (Hargreaves & Fink; 2006) and 
for amplifying technology-rich pedagogies (Chang, Chin & Hsu, 2008; Jacobsen, 2006; McGarr & Kearney, 2009; 
Reeves, 2009); however, principals often have limited personal and professional experiences with new curricula, 
new pedagogies and new technologies (Anderson & Dexter, 2005). As a result, it can be a challenge for principals, 
as instructional leaders and learners themselves, to shift mindsets towards knowledge building pedagogies that foster 
deep thinking, global knowledge and technological skills. The theoretical framework for the research study draws 
upon literature in three inexUicably connected areas: theory, pedagogy, and leadership. 

Knowledge Building Learning Theory. 

Research demonstrates that computers alone offer little benefit for teachers and students (Bransford, et ah, 
2000; Cuban 2001; Schacter, 1999). Further, computers paired with information transmission practices do not yield 
improved student achievement (Scardamalia & Bereiter, 2006). However, learning scientists have found that 
technology does benefit learning when closely connected with collaborative knowledge building pedagogies, and 
when the teacher, as facilitator, engages students in technology-rich experiences that lead to deep thinking and 
learning (Barab, Arici and Jackson, 2005; Sawyer, 2006, 2008). 

Transformative 21 s ' Century Learning Pedagogies. 

The disconnect between common school experiences and demands of a participatory digital world are clear 
(Jacobsen, 2010). Zhao (2007) calls to expand education beyond core, testable subjects to education that is more 
interdisciplinary, celebrates and sponsors diverse talents, including digital citizenship, in a global era. Merely using 
technology to support conventional practices and providing access to new tools is not sufficient to promote 21 st 
century learning; hence, it is necessary to contemplate how to effectively use technology to sponsor deep thinking 
and to foster collaborative knowledge building (Spires, Wiebe, Young, Hollebrands & Lee, 2009). There is a need 
for authentic engagement in collaborative inquiry and for deep thinking and learning; thus, we need transformative 
pedagogies (Dede, 2007a, 2007b; Friesen, 2009; Sawyer, 2006). 

Growth-oriented Leadership. 

Research demonstrates that school leaders require professional learning opportunities to increase awareness 
of their roles as instructional leaders in a digital age (Deryakulu & Olkun, 2009; Flanagan & Jacobsen, 2003; 

McGarr & Kearney, 2009). Professional learning can be a challenge due to limited resources and a shortage of 
experts who can effectively model growth-oriented leadership and transformative pedagogies for collaborative 
knowledge building. Consequently, alternative models for professional support for principals could include 
networking with other similar-sized schools and collaborating between schools to nurture professional dialogue and 
provide time for reflection regarding the possibilities for technology-rich teaching and learning (Fullan, 2010; 
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McGarr & Kearney, 2009). Networks will provide new opportunities, such as virtual environments, social 
networking technologies and other online professional learning communities, for growth-oriented leadership (Spires 
et al.. 2009). 

Research Design 

An explanatory, mixed method case study approach was employed for this study. Based on Creswell's 
(2009) ‘Sequential Explanatory Research Design', rich and diverse interview, observation and other qualitative data 
were collected in the 2 nd phase of the study to expand on quantitative survey findings from the 1 st phase of the study. 
Case study is a “strategy of inquiry in which the researcher explores in depth a program, event, activity, process, or 
one or more individuals” (Creswell, 2009, p. 13). Information was gathered from principals and teachers in three 
diverse school districts that were in the midst of innovative reform involving educational technology through an 
exploration of the actions of the principals over a period of one year. Both quantitative and qualitative data 
collection methods (selected-response and open-ended survey items) were followed by qualitative methods (semi- 
structured interviews, observations, artifacts). The primary research question was, “How do principals cultivate 
teaching and learning improvements integrating technology that meet the needs of today’s learners?” In order to 
answer the question, the research explored the following supporting questions: 

( 1) How do principals perceive their role in leading teaching and learning improvements integrating technology? 

(2) To what extent do social and technological networks support principals during an educational reform? 

(3) In what ways does the principals’ conceptualization of their leadership practice change during the diffusion of 
teaching and learning improvements integrating technology? 

(4) How do teachers describe the leadership actions needed to support teaching and learning improvements 
integrating technology? 

(5) How are principals managing the key challenges of planning, implementing and sustaining teaching and 
learning improvements integrating technology? 


Data Collection 

In 2009, Alberta Education released leadership dimensions in the Principal Quality Practice Guideline to 
acknowledge the instructional leadership role required of all formal school leaders and the skills necessary for 
leading schools in a rapidly changing world. The leadership dimensions and associated descriptors of daily practice 
imply principals are responsible for cultivating new ideas about learning, also known as 21 st century learning, and 
for supporting teachers in adopting new pedagogies in order to meet the needs of today’s learners. The online 
survey used in this study was primarily based on the seven interrelated leadership dimensions and descriptors of 
daily practice: (a) fostering effective relationships; (b) embodying visionary leadership; (c) leading a learning 
community; (d) providing instructional leadership; (e) developing and facilitating leadership; (f) managing school 
operations, and resources; and (g) understanding and responding to the larger societal context (Alberta Education, 
2009). The survey items invited principals to reflect on their daily practice and how they perceive their role in 
leading teaching and learning improvements integrating technology. Using a five-point scale, participants selected 
responses ranging from a level of “not performed” to a level of “proficiency” whereby a principal could teach the 
practice to others. Additional survey items were used to determine the extent to which principals report the use of 
social and technological networks to support professional learning. 


Table 1. Survey Response Rate 



n 

Responses 

Response Rate % 

Total Sample 
Jurisdiction 

152 

39 

26 

A 

86 

21 

24 

B 

36 

10 

28 

C 

30 

8 

27 


Following the survey (Table 1), semi-structured interviews were conducted with a total of 23 participants in five 
schools. Participants included principals, assistant principals and teachers. Guiding questions with thematic 
interview questions based on the literature review were used to understand how principals cultivate teaching and 
learning improvements integrating technology. During the interviews many participants also shared documentation, 
schools websites, blogs and two participants invited the researcher to participate in professional learning sessions to 
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collect additional field notes. The interviews were audio-recorded resulting in over 235 pages of typed single-spaced 
transcripts along with 140 pages of hand written, livescribe field notes. 

Participants were invited to attend an online focus group six months after the interview phase to review and 
validate the study findings. Two online focus group sessions were held. A total of four participants from two of the 
five schools participated in the online focus groups. During the online focus group the researcher reviewed the 
findings and asked participants to discuss the following questions: 

1 . What are the challenges for educators in fostering international skills and student growth towards a global 
mindset across the curriculum? 

2. What is needed at the school or division level to support collaborative partnerships among teachers and 
researchers in preparing students for a more technical and globalized world? 

3. What supports are needed to encourage growth and advance learning for the early adopters in school 
improvements integrating technology? 

4. How has your use of social or technological networks for professional learning changed during the past 
year? 

Participants were also asked to reflect on the past school year and to share a comment about the progress of the 
school project or innovations involving technology. 

Innovative Research Methods and Technology for Research 

Throughout the study, the researcher used a variety of technologies to support innovative research methods. 
The technologies used and the research applications are shown in Table 2. 


Table 2. Technologies for Research 


Tool 

Research Application 

SurveyGold Golden Hills Software 

Online survey 

Email 

Participant Communication 

Skype 

Participant Communication 

LiveScribe Smart Pen 

Audio Recording Interviews & Field Notes 

iPad 

Audio Recording Interviews 

Elluminate Live! 

Focus Group 


There were both benefits and constraints in using innovative research methods and technologies for 
research along with associated ethical considerations. An American software company, SurveyGold Golden Hills 
Survey Software Inc., was used for the secure, online survey. One benefit in using password protected software is 
the ease of use in designing the survey and entering selected-response and open-ended survey items. With minimal 
knowledge in HTML, it was possible to adjust formatting, add bold titles and insert hyperlinks to external 
documents for references. Three unique surveys, one for each jurisdiction, were created to personalize the survey 
and to allow for data aggregation by jurisdiction (as shown in Table 1). 

Surveys were administered during October 2011 to all the principals in the three school jurisdictions. Given 
that online surveys are easy to create and administer, school principals are asked to complete surveys on a regular 
basis. Principals in Jurisdiction A indicated that October was an extremely busy time for survey requests and many 
described survey overload or feeling survey fatigue as a possible reason for low survey completion rates. 

Email was used for ongoing communication with jurisdictions and participants. Jurisdiction B provided a 
list of all the principals which allowed the researcher to directly communicate with the participants via email. The 
researcher was able to send email notes and follow-up reminders regarding the availability of the survey which may 
have resulted in the slightly higher participation rate compared to the other jurisdictions as shown in Table 1. Both 
jurisdiction A and C provided a research liaison that forwarded the researcher’s email reminders to the principals. 

A principal from jurisdiction B met with the researcher using Skype, a voice and video web-based service, 
prior to the in-person interviews taking place at the school. The researcher had not previously met the principal so it 
was helpful to have a Skype meeting and establish a rapport prior to arriving at the school. In all other cases the 
researcher communicated with each principal via email prior to the in-person interviews. 

The Livescribe smart pen, slightly larger than a conventional pen, is a computing platform for linking audio 
to handwriting. The smart pen was used during the semi-structured interviews to record handwritten field notes in a 
dot paper journal and used as the backup audio recording device during the interviews. The written notes and 


43 



associated audio segments resulted in portable document format (PDF) files and were organized using the Microsoft 
OneNote application to allow the researcher to quickly access specific locations in the written notes and audio 
recordings. The researcher started each interview with a brief introduction of the smart pen and realized this was an 
unfamiliar technology for the majority of the interview participants. Consequently, the pen provided a great ice 
breaker during interviews which resulted in many questions about the smart pen and discussions about the potential 
use of the smart pen in the classroom. Interestingly, one of the participants allowed the researcher to record the 
interview using the smart pen but did not want to be interviewed using the primary audio recording application on 
the iPad. Perhaps the smart pen seemed less intrusive as a recording device. Also, in one case the researcher was 
unable to retrieve the audio recording from the primary recording device and used the smart pen recording to 
transcribe the interview. One of the constraints in using the smart pen is the potential for audio quality to be 
impaired as the recorder is extremely sensitive to external sounds and motion, such as the researcher writing with the 
smart pen during recording. Also, during one interview the researcher reverted to a traditional pen when the ink ran 
out in the smart pen. As a result, for one interview the researcher reviewed the handwritten field notes and audio 
recordings as separate files instead of the integrated Livescribe text-audio files. 

The Apple, Inc. iPad was used as the primary audio recording device for the in-person interviews. Similarly 
to the smart pen, the iPad was very easy to use, quick to start-up and the battery life was sufficient for a full-day of 
audio recordings. However, in contrast to the smart pen, the quality of the audio recording with the iPad was 
superior in clarity with little to no interference from sounds such as the researcher writing with a pen or participants 
sitting further away from the recording device. During one interview the recording on the iPad stopped unexpectedly 
and approximately 40 minutes of recording did not save. However, the researcher was able to retrieve the recording 
for that period from the smart pen in order to transcribe the complete interview. 

Elluminate Live, a web conferencing application through Blackboard Inc., was used for the online focus 
group. All interview participants were invited to participate in one of two scheduled web conferences during May 
and June 2012 or could provide feedback to the researcher using email. Many participants expressed enthusiasm 
about using web conferencing technology for a follow-up meeting to validate findings. Three of five schools agreed 
to participate in the online focus group. Participants from two of the schools were able to successfully connect to the 
Elluminate Live sessions, respond to the overall findings and provide feedback. The Elluminate Live sessions were 
recorded capturing audio and text interactions as well as the slide deck used to display visuals during the session. In 
addition, the researcher recorded the audio with the iPad recorder. 

According to Murthy (2008) blogs have potential for rich interactivity but it is important to consider the 
public nature of blogs and the issue of protecting the anonymity of participants. The researcher initially thought a 
blog or micro blogging space could be used to engage participants in anonymously commenting on discussions 
about the research. Any information provided on the blog would keep the identity of participants confidential 
through anonymous postings. However, in micro blogging spaces, such as the Twitter service, the identity of the 
participants is not confidential. Unexpectedly, the researcher found that reviewing publicly accessed twitter feeds 
from participants in the study, or other personnel from the respective jurisdictions provided additional insight 
regarding the professional learning activities occurring in the jurisdiction, innovative projects and areas of 
educational focus. 

It was necessary to consider ethical implications for using various technologies for research purposes. For 
instance, in the opening of the survey the participants were provided with general information about the survey 
questions being subject to U.S. laws, including the USA Patriot Act. In addition, there was a reminder about the 
risks associated with participation being minimal, however, and similar to those associated with many e-mail 
programs and social utilities spaces. Participants were also asked to agree to the terms provided and consent to 
participation before moving to the actual survey questions. In the online focus group, the visual and named identity 
of participants was kept confidential by using assigned pseudonyms. Still, absolute anonymity or confidentiality 
cannot be guaranteed as other individuals could potentially recognize participant’s audio contributions. 

Overall, the use of technology for research had many benefits such as using email and web-based Skype 
service for providing efficiencies in communicating with participants. Another benefit included using an online 
survey and a web conferencing application for collecting data from participants across three jurisdictions located in 
different areas of the province. Modeling use of technologies, such as the smart pen and iPad during the interviews, 
provided a common starting point for all the interviews and an interesting discussion topic that appeared to help 
interview participants quickly feel at ease with the researcher before responding to the specific interview questions. 
Lastly, the technologies supported the researcher in the analysis process. Digital files connecting the handwritten 
field notes and audial contributions facilitated the analysis process as the researcher was easily able to locate and 
review sections of the interviews by selecting the hyperlinked text and at the same time retrieving the associated 
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audio from the interview. Other technologies were also used minimally by the researcher for general 
communications about the research, such as the researcher's blog and Twitter feed. 


Antecedents for Educational Technology Leadership 

The rich and diverse range of data from the surveys, interviews, qualitative documentation and researcher’s 
field notes were interpreted in light of information from the literature as a means to substantiate the emergent 
themes, evidence, processes and perspectives. All data was validated and clarified through triangulation by the 
rigorous deployment of a sequential explanatory research design using mixed data sources. An emerging code 
perspective was taken when reviewing the qualitative data where the codes were not predetermined and emerged 
during the data analysis. As a result, several themes emerged as antecedents for educational technology leadership, 
including visionary leadership for innovation; research-informed professional learning; attention to contexts for 
support; monitoring for continuous learning and growth; and creativity and openness for change. 

Visionary Leadership for Innovation. 

Effective technology leadership requires leaders to develop and articulate a vision for innovation and 
change (Chang, et al„ 2008; Hew & Brush, 2007; Wagner, 2003; Yu & Durrington, 2006). The Change Leadership 
Group at Harvard’s graduate school of education focuses on systemic improvements in schools and districts and 
identifies the development of a shared vision as central to instructional improvements (Wagner, 2003). Likewise, the 
interview participants echoed the importance of visionary leadership for meaningful technology integration and 
working towards a shared vision in the community. Participants identified resources, such as the National 
Educational Technology Standards for School Administrators (NETS for Administrators, 2009), used to support 
leaders in developing a shared and technology-infused vision for instructional leadership. Principals perceived 
themselves as demonstrating visionary leadership, with close to 80% of all principals surveyed indicating they 
embody visionary leadership approaches and consistently ensure that planning, decision making, and 
implementation strategies are based on a shared vision and an understanding of the school culture in the context of 
technology integration. However, in contrast, researchers recently found visionary leadership for learning with 
technology limited and described as a barrier in a recent two-year study examining the relationship of technology 
use, student engagement and student success in high schools across the province (Alberta Education, 2010c; 
Jacobsen, Friesen, Daniels & Varnhagen, 2011). 

Research-informed Professional Learning. 

Professional learning can serve to influence attitudes as well as build knowledge and skills pertinent to 21 st 
century learning. School leaders require opportunities for increased awareness of their roles relative to educational 
technology (Deryakulu & Olkun, 2009; Flanagan & Jacobsen, 2003; McGarr & Kearney, 2009). Similarly, 
researchers suggest limited time for professional dialogue, and ongoing professional development can be barriers 
that restrict technology integration (Alberta Education, 2010c). The good news is that close to 95% of the survey 
respondents indicated that the school principal promotes and facilitates meaningful professional development for 
teachers and other staff in the context of technology integration. It is also important for principals to participate in 
professional learning involving educational technology to understand the instructional changes required for teaching 
and learning with technology. Fullan (2008) argued that fostering continuous job-embedded learning as one of the 
secrets of change to ensure learning at work is part of daily work for everyone including leaders at all levels of the 
organization. Principals interviewed did not perceive themselves as possessing a high level of technological fluency 
but did discuss the value of professional learning; however, the teachers from the same school described the 
influence of the principals in adopting new technologies and perceived the principals with a high level of proficiency 
in integrating educational technologies. So, while the principals did not perceive themselves holding sufficient 
technological competency, they agreed it was a necessary condition for effective technology integration and value 
research-informed and ongoing professional learning. Many principals discussed their voluntary participation in this 
study or other research and grant-funded projects as rich opportunities for technology-related professional learning. 

Chang et al. (2008) defined a technology leader as “one who leads the school in improvement or restructuring, 
and uses emerging technologies as the core resources for educational change” (p.241). One of the results in Yee’s 
(2001) investigation of Canadian, U.S. and New Zealand principals’ experiences in technology-rich schools found 
that principals need to develop personal competency and would value professional learning opportunities. Dawson 
and Rakes (2003) conducted an exploratory study with K-12 principals and found those involved in long term 
technology-curriculum integration training significantly influenced the level of technology use at the school. 
Similarly, the interview participants conveyed the importance of professional learning and suggest principals and 
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teachers involved in long-term technology integration projects with access to professional learning influence the 
level of technology use at the school. 

Contexts for Support. 

Fostering adaptation of instructional improvements for 21 st century learning requires attention to context. 

Zhao, Pugh and Sheldon (2002) defined context as the (1) human infrastructure, (2) technological infrastructure and 
(3) social support. First, human infrastructure, refers to the people in the organization providing support for the 
innovation and includes on-site support as well as off-site experts that can be accessed (Jacobsen, 2006). Second, 
technological infrastructure is another aspect that continues to provide challenges for schools and districts and 
requires leaders to “develop robust conceptions of technology in education,” and to allocate appropriate resources to 
impact student learning (Shuldman, 2004, p. 338). Third, social support among colleagues is best described by 
Fullan (2008) as leaders fostering “purposeful peer interaction,” (p. 12) which is another one of his secrets of 
change. Similarly, in this study, the interview participants suggested the three key areas for support when integrating 
technology in schools include human infrastructure, technological infrastructure and social support. Flowever, very 
few participants discussed the use of technological networks for social support and instead relied on the support of 
colleagues at the school. The level of human infrastructure or having key staff members with technological fluency 
was not a concern for the schools participating in the interview phase of the study. Many schools have technology 
mentors or several teachers on staff assigned to school leadership roles in supporting the use of technology. For 
instance, in some schools, the principals designated a technology lead for each grade level. The participants all 
indicated there were individuals at the school that could provide the necessary professional development support in 
using technology. Although, as described in the diffusion of innovation model (Rogers, 2003), there is a range of 
acceptance and use when considering the levels of adoption of any new innovation. The participants indicated there 
is a continued focus on the needs of the mainstream and later adopters while inadvertently neglecting the needs of 
the innovators or early adopters. For example, jurisdictions create “how to” documents and emphasize the ease of 
use for district approved technologies and in some cases prohibit or limit the use of emerging or more difficult-to- 
use technologies. The early adopters are generally the ones developing arguments or cases for access to more 
advanced and complex applications. 

The participants also described a robust technological infrastructure in the school providing a variety of 
opportunities for students and teachers to access different types of technology throughout the day. Some participants 
indicated that due to the increased use of the Internet, the bandwidth could be improved for more efficient access to 
information and resources. Some participants discussed the challenge for technicians to keep abreast of new 
technologies and to access appropriate training to support schools when purchasing new-to-the-market technologies. 
In these cases the participants discussed the issue of feeling isolated when troubleshooting technical difficulties or 
setting up newer technologies for school use. 

The technological infrastructure at the schools continues to grow each year as does the rate of usage by the 
teachers and students. For example, in the past, the early adopter teachers recall having unlimited access to labs of 
computers and appreciating the opportunity to use technology whenever needed. One of the challenges the early 
adopter teachers are finding is that access to the technology has actually decreased as their colleagues become more 
technologically fluent and want increased access to the resources. The decreasing level of access to technology is an 
unexpected challenge for teachers who previously had ready access and in some cases unlimited use of school 
technologies. Perhaps as the adoption of student-owned devices increases in the schools, the level of access to 
technological tools for teaching and learning will not be a concern. 

Moreover, professional development is limited for those with advanced technological fluency. Research shows 
teachers are interested in transformational practices integrating technology and require more effective professional 
learning opportunities emphasizing pedagogy instead of technology training (Alberta Education, 2010c). The NETS 
for administrators (2009) suggests digital age leaders “facilitate and participate in learning communities that 
stimulate, nurture and support administrators, faculty, and staff in the study and use of technology” (p. 1). Similarly, 
participation in professional learning communities was described as a common form of professional development by 
participants in the study (Dufour, Dufour & Eaker, 2008). Interview participants, who identified as early adopters, 
expressed the need to be challenged to extend their technological skill set and find it difficult to collaboratively plan 
or share practice with others in the professional learning communities comprised of teachers with varying levels of 
technological fluency. There are limited professional learning opportunities available at the advanced level 
necessary for early adopters; most professional learning sessions aim to the needs of those with entry level skills. 

Likewise, principals across all three jurisdictions in the study expressed concerns about being the only one or 
one of a small group of principals in the school jurisdiction undertaking a similar technology integration project. 

The findings suggest principals are aware of emerging technologies and desire increased proficiencies for 
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themselves and for the teaching staff. The principals shared various methods used for support with professional 
learning related to innovations and new school projects, such as social and technological networks. The social 
networks described were either formal professional learning communities with set dates for meetings and a set 
membership or informal group meetings with a small group with similar interests. The technological networks were 
not discussed as frequently as the social networks by the interview participants. One of the principals interviewed 
shared her extensive use and reliance on technological networks for professional growth and keeping abreast of 
educational and technology changes. She regularly posts to a blog site where she shares her professional growth 
plan and learning throughout the year; she also uses Twitter on a daily basis to connect globally with principals and 
other leaders in education. This principal has always relied on mentors external from the human supports in her own 
school to continue to build knowledge and understanding about learning and leading but has now extended her 
mentorship circle to purposeful peer interactions with online mentors in digital spaces. 

Monitoring for Continuous Learning and Growth. 

Supervision includes ongoing monitoring and research to promote investment in educational innovations. 
Wagner (2003) used the term supervision to describe monitoring that is frequent, rigorous, and focused on the 
improvement of instruction. He also stressed supervision is “conducted by people who know what good instruction 
looks like” (Wagner, 2003, p.28, 30). The Galileo Educational Network Association is an excellent example of a 
professional development and research organization focussing on supporting all levels of the educational system at 
once. Professionals work with educators and leaders in providing current research and supervision of innovations to 
promote continuous learning and growth (Jacobsen, 2006). The principals interviewed all described a need for 
increased support from research partners to leant from their own educational innovations or projects. All schools in 
this study included participants who were previously involved in a three-year grant funded one-to-one laptop 
initiative in the province (Alberta Education, 2010a). The participants value opportunities to learn from themselves 
by participating in research projects and continue to seek practitioner-researcher partnerships. 

Creativity and Openness for Change. 

Zhao (2012) advocates leaders and educators increase emphasis on creativity and entrepreneurism in order 
to stimulate the diversity of talent in schools and to be globally competent. There are varying degrees to which 
schools foster entrepreneurism or “the desire to solve problems creatively” and there a decline in creativity in 
schools (Zhao, 2012, p. 9). The leadership dimensions in the Principal Quality Practice Guideline (2009) and the 
NETS for administrators (2009) both include descriptors related to global issues; however the documents lack 
descriptors related to creativity or entrepreneurial skills for digital-age leadership. Interestingly, the interview 
participants discussed many ways in which principals are modeling creativity and used words to describe principals 
similar to those found in Zhao’s (2012) description of creators: “Creators are curious people, who keep wondering 
and imagining. Creators are confident people, who are courageous to think and act outside the box. Creators are, 
well, creative people who can come up with novel ideas and solutions. . . . They must be allowed the freedom and 
encouraged to wonder and wander, to explore, and to experiment” (p.239). 

Furthermore, the participants provided a variety of examples demonstrating creative principals in action. 
For example, principals are transforming staff meetings from sharing important information to gatherings for 
professional dialogue, learning and growth; shifting morning announcements from typical adult-led intercom audial 
messages to student-led audio-visual broadcasts focussing on inquiry; using social media including blogs and 
microblogs to communicate beyond the school community; pioneering uses of new technological tools for 
personalized learning, such as the iPad, in advance of other schools; finding creative ways to solve problems, such 
as providing each student with a Boogie Board for school wide initiatives to reduce paper waste; and making 
provisions for global partners and global connections, to name only a few. Principals in this study demonstrate 
creativity and openness for change in solving problems and preparing students for a technical and globalized world. 

Analysis revealed similarities and differences in how leaders enact change as well as unique and common 
challenges and issues presented by multiple participants and multiple sites. Several prevailing themes emerged as 
interrelated antecedents for educational technology leadership, such as, visionary leadership for innovation; 
research-informed professional learning; attention to contexts for support; monitoring for continuous learning and 
growth; and creativity and openness for change. Evidence was found that school leaders are concerned with 
preparing students for a more technical world; however, little evidence was found that leadership actions responded 
to a larger societal context, or fostered international skills and growth towards a global mindset. From a 
methodological perspective, findings suggest the need for school leaders to make provisions for collaborative 
partnerships among teachers and researchers in preparing students for a more technical and globalized world, that is, 
research converging theory and practice in educational contexts known as design based research (Dai, 2012; 
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Gersalfi, Barab & Sommerfeld, 2012). Overall, the findings from the research suggest there is a need for leaders to 
build digital-age skills, optimize creativity, entrepreneurship and foster a knowledge building environment 
promoting continuous innovation and growth. 

Discussion and Recommendations 

The following key recommendations from the study are: leaders need digital-age competencies; there is a 
need for technology-enhanced learning environments for ongoing professional learning; and design based research 
should be considered for increasing practitioner-researcher partnerships and cultivating innovation in schools. 

The role of school leaders, such as the school principal, is changing and as a result the competencies 
expected of effective school leaders need to change respectively. A recent initiative in Alberta, Inspiring Education, 
describes the vision of a 21 st century student as an engaged thinker, ethical citizen and entrepreneur (2010b, p.5-6). 
Similarly, Zhao (2012) supports increased attention to creativity and building an entrepreneurial spirit. Defining 
today’s learner provides a foundation for learning and leading; however, documents outlining expectations or 
competencies for leaders, such as the Principal Quality Practice Guideline (2009) leadership dimensions in Alberta 
inadequately emphasize the need for leaders to build digital-age skills, optimize creativity, entrepreneurship and 
foster a knowledge building environment promoting continuous innovation and growth. Many studies (Alberta 
Education, 2010a, 2010c) focus on the benefits of technology-enhanced learning environments and highlight the 
need for increased professional learning opportunities needed for teachers to improve pedagogies with new 
technology. Moreover, the findings from this study suggest school principals need digital-age competencies for 
instructional leadership. 

Although there are limitations to the study, such as the sample size involved and the length of the study, the 
findings in the study correspond and also build and extend upon findings in the literature where principals are 
identified as key leaders for amplifying technology-rich pedagogies (Chang et al., 2008; Jacobsen, 2006; McGarr & 
Kearney, 2009; Reeves, 2009) and maintain that support for educational technology leadership is necessary in 
schools for preparing students for a more technical and globalized world. It is also recognized there are many formal 
and informal leadership roles necessary for cultivating teaching and learning improvements integrating technology 
in education and this study specifically focussed on principals in schools. Based on the findings of this research, it is 
recommended that school jurisdictions support the inclusion of digital-age competencies for leaders when investing 
in instructional leadership. 

Ongoing professional development is a barrier in Alberta (Alberta Education, 2010c; Jacobsen et al., 201 1) 
and as new technologies are leveraged in schools, professional development models for supporting teachers and 
administrators with their work need to undergo changes. Instructor-led professional development held afterschool at 
defined locations can be accessed asynchronously as self-directed, online sessions at a convenient time for learners. 
Many participants discussed the formation of professional learning communities as a common strategy for 
advancing skills and professional learning support for school improvement (Dufour et al., 2008). In contrast, at the 
time of interviewing participants, very few teachers or administrators in the study were utilizing social media or 
technological networks as a means for building professional connections and advancing professional learning. A 
future study may be considered to explore how social and technological networking can be used for professional 
learning and to support professional learning communities. 

Study findings should be read with the knowledge that the participating schools were all highly involved in 
planning, implementing and sustaining technology integration projects and were interested in sharing their 
experiences. Moreover, the principals in the schools were described as risk-takers and creative. Perhaps, school 
leaders with limited interest in technology integration or not involved in projects or innovations involving 
technology may not have volunteered to complete the survey even though participation was completely anonymous. 
A commonality among the principals interviewed was a desire for research-informed decision making at the school. 
The principals shared a variety of examples, including volunteering for this research study, to demonstrate taking 
initiative and advocacy for participation in research projects involving technology integration. The school leaders 
recognize the benefits of working in collaboration with researchers to advance understandings of theory and practice 
when involved in designing interventions or innovations in learning. A common barrier discussed by participants 
was the lack of time for professional dialogue and working collaboratively with peers which is consistent with 
findings from other studies (Alberta Education, 2010c; Jacobsen et al., 201 1). A recommendation resulting from this 
study is to consider the model of design based research (Dai, 2012; Gersalfi et al., 2012) as a method of study during 
school time for increased collaboration between practitioners and researchers valuing the interdependency between 
theory and practice. 
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The findings and experiences from this study are useful and can be highly influential in the design and 
delivery of distributed professional learning experiences for school principals. Findings will be of particular interest 
to educational technology researchers, leaders in education departments, school authorities and school districts, 
administrators in school districts, and those scholars and practitioners interested in leadership professional 
development and those persons interested in the design, implementation and assessment of technologies for teaching 
and learning used in schools. 

It is important to recognize that little change in education can take root or even flourish unless principals 
have a key role in planning, implementing and sustaining teaching and learning improvements integrating 
technology in order to prepare students for a globalized world. Principals from three diverse school districts across 
Alberta were surveyed in order to explore their perceived role in cultivating teaching and learning improvements 
integrating technology and how social and technological networks impact leadership during educational change. 
Administering a broad quantitative survey, followed by semi-structured interviews with principals and teachers and 
collecting additional qualitative documentation, provided a powerful approach in conducting an in-depth 
explanatory case study of principals involved in educational change in K-12 schooling environments. This case 
study will serve to amplify the actions of principals involved in cultivating teaching and learning improvements 
integrating technology that meet the needs of today’s learners and the use of innovative methods and technologies 
for research purposes. 
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Abstract 

This study compared student perceptions of using a traditional linear discussion board to using a spatial 
discussion board for online course discussions. In a two factor wi thin-subject experimental design, graduate students 
participated in a series of online discussion activities, alternating between posting in the linear discussion board and 
the spatial discussion board. They were surveyed about their attitudes regarding the formats. Generally, the familiar 
linear format was preferred, although attitudes about the spatial format increased over time. 

Introduction 

Linear text discussion boards are a common tool in online courses. They are often used with the goal of 
increasing student-student interaction (Carr-Chellman & Duchastel, 2000). The constructivist learning theory 
emphasizes knowledge construction of individuals and of social interactions (Driscoll, 1994). Vygotsky's social 
interaction theory similarly states that social interactions are important for learning (Vygotsky, 1978). Therefore, 
one could assume that student-student interactions on discussion boards can conUibute to learning. 

Unfortunately, linear text discussion boards have not been as effective as these theories would predict. 
Students often see posting to the discussion board as superfluous and inconvenient (Lapointe & Reisetter, 2008) or 
just as an assignment instead of an opportunity to engage in discussion with their classmates (Aleksic-Maslac, 
Magzan, & Juric, 2009). Several studies have looked at ways to improve the quality of online discussions, such as 
increasing internal motivation (Bennett & Monds, 2008) and encouraging debate (Kanuka, Rourke, & Laflamme, 
2006). But instead of Uying to make linear text discussions more effective, perhaps an alternative format would lead 
to better discussions. 

Spatial representations of information can be beneficial to learning. For example, visual displays help 
students group information into schemas, free up working memory for other thought processes, and help students 
see visual patterns that would be hard to understand in a sequential language sttucture (Hegarty, 201 1). A map can 
integrate information, presenting related concepts in the same visual space instead of the information being scattered 
throughout written text (Nesbit & Adesope, 2006). For example, several students may have similar comments or 
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opinions, but on a linear text discussion board, their posts are dispersed throughout the thread. With a spatial format, 
similar ideas are placed in the same visual location, which helps students see at a glance the distribution of opinions 
among the class. 

This study explored the following research questions: 

1. How do students’ perceptions differ between linear and spatial discussion formats in regards to ease of 
use, perceived usefulness, social presence and attitude? 

2. How do student’s perceptions of each format change over the course of the semester? 

Method 

Participants 

To address the above research questions, we collected data from graduate students in an introductory level 
instructional technology course at a large southwestern university. Twenty students completed at least some of the 
surveys, but only 1 1 students completed all surveys and were included in the analysis. 

Discussion Boards 

The course was held entirely online over the course of a fall semester. During the semester, students were 
asked to respond to eight discussion topics; students discussed each topic for one week. The instructor provided a 
topic or question and students responded using either a linear or spatial discussion board. Students were required to 
create an original post and respond to at least two peers. Using a within groups design with counterbalancing, 
students alternated which discussion board they used, so that by the end of the semester, they had participated in 
each format four times. Prior to participating in graded discussions, students posted introductions to both discussion 
formats in order to familiarize themselves with both systems. 

This study used the Blackboard content management system for the linear discussion board as shown in 
Figure 1. Students’ original posts are left aligned, with responses indented. 
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Figure 1. Linear Discussion Board (Blackboard) 
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MindMeister ( http://www.mindmeister.com/ ) was used for the spatial discussion board (See Figure 2). 
MindMeister is an online mind mapping tool. Students placed their responses visually along a continuum to indicate 
their agreement or disagreement with the topic. Responses branch out to the sides. 
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Figure 2. Spatial Discussion Board ( MindMeister ) 
Data Collection and Analysis 


After each discussion, students completed a survey by indicating their level of agreement with fifteen 
Likert-type statements (5-point scale). The survey items were designed to measure four constructs 1) ease of use, 2) 
perceived usefulness, 3) social presence and 4) attitude. After discussions 1, 2, 5, and 6, students were also asked 
three open-ended questions: 1) What did you like about the discussion tool (i.e.. Blackboard or Mindmeister)? 2) 
What challenges did you experience when using the discussion tool? and 3) What instructional advantages do you 
see in using the discussion tool? 


Results 

The first research question asked “How do students’ perceptions differ between linear and spatial 
discussion formats in regards to ease of use, perceived usefulness, social presence and attitude?” See Table 1 for a 
summary of means and standard deviations on these measures. 
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Table 1. Means and Standard Deviations for all measures. 


Discussion Format 
( n= 11) 

Linear Discussion Board 

Spatial Discussion Board 

Perceptions 

First Use 

Last Use 

First Use 

Last Use 

Ease of Use 

4.79 (.22) 

4.75 (.47) 

4.00 (.74) 

4.15 (.85) 

Usefulness 

4.33 (.71) 

4.36 (.81) 

3.63 (.86) 

4.06 (.94) 

Social Presence 

4.27 (.99) 

4.24 (.54) 

3.79 (.81) 

4.21 (.76) 

Attitude 

4.52 (.69) 

4.42 (.50) 

4.09 (.93) 

4.39 (.81) 


*Standard deviations are presented in parentheses 


To further examine this question, a two-factor within subjects ANOVA was conducted, with discussion 
format (linear vs. spatial) and time of use (first use vs. last use) as the within-subjects factors. Significant difference 
of discussion format were found favoring the linear discussion format for ease of use, F(l,10)=l 1.145 ,p=.008, 
perceived usefulness, FT 1 , 1 0)= 1 4.559, p=.003, and social presence, FT 1 , 10)=5.92, p=. 043. No significant results 
were found for attitude,/’) I , 10)=3.6 1 7, /;=.086. Regarding the effect of time of use, no significant differences were 
found for the time of questionnaire administration (first use vs. last use) nor was the interaction effect significant. 

The second research question asked “How do student's perceptions of each format change over the course 
of the semester?’’ Dependent t-tests were conducted. For the linear discussion board, there was no significant 
change. For the spatial discussion format, there was a significant change on social presence, f(10)=-2.971, p=.0 1 4, 
although attitude approached significance, t( IO)=-2.l93,/;=.053. 

When comparing differences in perceptions between the two formats at the beginning of the semester using 
dependent t-tests, significant differences were found across the board, favoring the linear discussion board: ease of 
use, f(10)=-3.480, p=.006, perceived usefulness, t( IO)=-3.2()2, /;=.009, social presence, t( 10)=-2.951, /;=.() 1 4, and 
attitude, t(10)=-2.971,p=.015. However, comparisons at the end of the semester show significant differences only in 
ease of use,f(10)=-2.319, /;=.043, still favoring the linear format. 

Discussion 

Overall, students preferred the linear format over the spatial format. A review of responses to the open- 
ended questions suggests that the familiarity of Blackboard is a contributing factor to this preference. 

“I liked discussing on the Blackboard because I was already familiar with the system.” 

“I use it [Blackboard] in all of my course online.” 

Exploring preferences at the beginning and the end of the semester, students’ opinions of the linear format 
did not change over time, but their opinion of the spatial format improved in feelings of social presence, and 
approached significance in attitude. Comparing the two formats at the beginning and end of the semester shows a 
stark preference for the linear format at the beginning of the semester, but only a significant preference in ease of 
use later on. Student comments indicate that as they became more familiar with the spatial format, they had more 
positive things to say. 

“It [MindMeister] gives me a sense of community with my classmates.” 

“I really enjoy the physical layout of the map.” 

“I also enjoyed being able to see the big picture.” 

There were also comments that showed a preference for the spatial format, especially for specific types of 
learners or discussions. 

“The discussion tool [MindMeister] created a visual flow of the discussion that a regular discussion board 

lacks.” 

“[MindMeister] helps to collaborate and link ideas well.” 
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By the end of the semester, there was little preference for one format over the other, suggesting that the 
differences may have been due to familiarity with the format. Students had used Blackboard in previous classes and 
linear discussions are used in other online discussion formats, such as forums and blogs. MindMeister was new and 
visual spatial tools are less common in other online discussions. Comments from students indicate that there were 
features of the spatial format that they preferred over the linear format. Echoing Nesbit and Adesope (2006), one 
student commented that the "visual layout is useful to see how different discussion posts connect and relate to each 
other." Perhaps with more comfort and familiarity with the tool, students will take more advantage of the benefits 
visual discussion formats offer. Future research in this area should control for the familiarity of the tool in order to 
explore the potential benefits of spatial discussion boards. 
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Abstract 

This study will investigate how learners’ emotional state by external mood induction influences learners’ 
engagement and learning performance. Previous studies suggest that emotions consist of valence 
(pleasant/unpleasant) and arousal (calm/arousing). Based on a motivated cognition perspective, this study expects 
that valence and arousal in multiple levels will differently influence motivated cognitive processing of learning 
content. Data will be analyzed to examine how valence and arousal interact with engagement for learning and 
retention test scores. The results will contribute to the understanding the effective use of emotions in instructional 
design. 


Introduction 

Emotions are universal and innate. Emotions can be generated or initiated in response to a stimulus 
including learning material (Desmet, 2002). A common view of emotions posits that emotions consist of two 
dimensions, valence (pleasant/unpleasant) and arousal (calm/arousing) (Russell, 2003). Pleasant and unpleasant 
stimuli produce positive and negative emotions, respectively. Calm and arousing stimuli cause activating and 
deactivating emotions (P. J. Lang, Bradley, & Cuthbert, 1997). Past studies found that positive emotions enhance 
motivation to learn, learning strategies, and learning outcomes (e.g., Goetz, Pekrun, Hall, & Haag, 2006; Pekrun, 
2006; Pekrun, Goetz, Tiltz, & Perry, 2002). Other studies found that arousal can improve memory and learning (e.g., 
Falk & Gillespie, 2009). However, in the previous studies, the effects of negative emotions have been ignored or 
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underestimated in educational settings. Moreover, the past studies did not consider the interaction effects between 
valence and arousal on learning. Media psychology studies have found that valence and arousal in multiple levels 
differently influence the amount of cognitive resources allocated to processing information (e.g., A. Lang, 2006). 
Based upon these studies, we have research questions as follows: 

(a) In calm levels of stimuli, will pleasant stimuli lead to higher levels of engagement for learning and higher 
retention test scores? 

(b) In arousing levels of stimuli, will unpleasant stimuli lead to higher levels of engagement for learning and 
higher retention test scores? 


Relevant Literature Review 


Emotional-Motivational Systems 

According to the motivated cognition perspective, people have emotional-motivational systems, which are 
appetitive and aversive systems (Cacioppo & Gardner, 1999; A. Lang, 2006). Environmental opportunities (e.g., 
food and mates) and threats (e.g., danger and death) are pleasant and unpleasant stimuli, respectively. Pleasant and 
unpleasant stimuli elicit positive and negative emotions. Arousal levels of the stimuli reflect the intensity of the 
motivational system activation levels. 

Motivated Cognition 

The appetitive and aversive systems have different functions of approaching pleasant stimuli and avoiding 
unpleasant stimuli (A. Lang, 2006). In an opportunity environment, people have the curiosity to remember as much 
about pleasant stimuli as possible. Therefore, under low levels of arousal (i.e., calm) condition, the appetitive system 
tends to allocate the more amount of cognitive resources allocated to processing information than does the aversive 
system. The greater tendency of appetitive activation is referred to positivity offset. 

Under high levels of arousal condition with a threatening environment, on the other hand, people try to 
identify as much information about unpleasant stimuli as possible (A. Lang, 2006). As arousal levels increase, the 
amount of cognitive resources allocated to processing the unpleasant information radically improves so as to protect 
from any potential danger or threat, when compared to the increase rate of appetitive activation. This steeper slope 
of aversive activation is referred to as negativity bias. However, when unpleasant stimuli are extremely life- 
threatening, resource allocation switches to retrieving escaping information to avoid the worst, such as death. 
Subsequently, the amount of cognitive resources allocated to processing external information abruptly falls down. In 
academic settings, life-threatening stimuli may be careful or inappropriate. Therefore, in this study, we focus on 
calm and arousing levels of pleasant and unpleasant stimuli in the low-to-moderate range of arousal. 

Emotion and Learning 

Several previous studies examined the effects of emotions in learning. Isen and Reeve (2005) found that 
interesting tasks (a three-dimensional puzzle) promoted college students' positive emotions, motivation to leant, and 
enjoyment of the tasks, when compared to uninteresting tasks (identifying strings of letters in correct alphabetical 
orders). Falk and Gillespie (2009) showed that the visiting experience at a science center (Goose Bumps exhibition) 
triggers K-12 students’ positive and arousing emotions, and simultaneously improved memory and comprehension 
of the science. Um, Plass, Hayward, and Homer (201 1) suggested that positive emotions can be induced by two 
ways: the design of learning content itself (e.g., layout, color, or sound (Wolfson & Case, 2000)) and external mood 
induction (e.g., showing films or giving free gifts before learning tasks (e.g., Isen, Daubman, & Nowicki, 1987)). 
However, there has been little research on the role of aversive activation on the high arousal condition. Thus, this 
study aims to investigate the interaction between emotional valence and arousal on online learning. 

Data Collection 

This study will use a 2 (valence: pleasant and unpleasant) x 2 (arousal: calm and arousing) experimental 
design. The treatment in this study will be external mood induction, which contains emotional video clips with 
various levels of valence and arousal. 

Pilot Study 

A pilot test will be conducted to choose four different emotional stimuli (calm pleasant/arousing 
pleasant/calm unpleasant/arousing unpleasant). Sixteen two-minute video clips will be chosen (e.g., peaceful 
landscape for calm pleasant, big winner in a sports game for arousing pleasant, big loser in a sports game, and terror 
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for arousing unpleasant). The 16 video clips will be presented to 30 undergraduate students in educational 
technology classes at a large southwestern university. After each video clip, the students will be asked to rate their 
emotions using self-reported emotional measures for positive valence, negative valence, and arousal. After 
completing the pilot test, an ANOVA test will be conducted to determine four emotional video clips best 
representing the four emotional categories. 

Participants and Procedures 

To address the above research questions, 120 undergraduate students will be recruited from a large 
southwestern university. They will be randomly assigned to one of the four emotional categories. This study will 
consist of four phases as follows; 

• First phase: emotional video clip (treatment) 

• Second phase: online learning content about the topic of public relations 

• Third phase: survey 

The participants will be asked to report their engagement levels towards learning contents. 

• Fourth phase: assessment 

The participants will be asked to take a retention test. 

Dependent Variables 

Engagement for learning will be adopted and modified from the one used in previous studies (e.g., Isen & 
Reeve, 2005). It will be measured using an eight-item seven-point Likert scale ranging totally disagree to totally 
agree. Then, retention of the learning content will be assessed by 10 multiple-choice questions. 

Data Analysis 

To answer the research questions, 2x2 between-subjects ANOVA tests will be computed with valence and 
arousal as two independent variables and engagement and retention test scores as data collection. 

Future Works 

We will conduct the pilot test to determine four emotional video clips in March. Data for this study will be 
collected in April and May. The findings will be presented at the conference with more detailed information. 
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Abstract 

In order to collect information about the efficiency of the use of self-regulated hyperdocuments in the acquisition 
of new learning an empirical study was designed to answer three guiding questions: 1) Is the hyperdocument 
efficient on students' achievement on problem solving tasks and the developed cognitive processes? 2) What is the 
scaffolding efficiency of the hyperdocument on the cognitive processes developed during problem solving? A 
qualitative study, based in a case study model, was carried out with eight 6th grade students. Firstly we developed an 
interactive hyperdocument based in self-regulated learning and problem based learning. The participants were 
involved in a problem solving task before exploring the hyperdocument and again after it. The data gathered through 
think-aloud protocols were compared. Results suggest advantages in the use of the hyperdocument, supporting the 
importance of the use of interactive and technological platforms in promoting self-regulation in students, thereby 
promoting more motivation, interaction, comprehension and achievement. 

Keywords 

Problem based learning, self-regulated learning, hyperdocument, think aloud. 

Introduction 

We are presently witnessing a technological revolution operating in Portuguese schools. Due to the 
Technological Plan for Education, schools are being equipped with technological resources that enhance a larger 
student involvement in the educational process. Aware of the influence technology has on students achievement 
(Waxman et al., 2003), and pursuing the objectives defined in the new Math Program for Middle School (ME, 
2007), we developed an empirical study based in the use of a hyperdocument about problem solving. To develop 
that hyperdocument we followed the principles of Self-Regulation in learning (Azevedo et al., 2005, 2007a, 2007b; 
Rosario, 2004; Rosario et al., 2006, 2007, 2008; Zimmerman, 1989, 2000; Zimmerman & Kisantas, 1997; 
Zimmerman & Risemberg, 1997) aiming to promote students autonomy. This concept sustains conceptually the 
hyperdocument we developed and subsumes the empiric investigation performed around the learning strategies, the 
metacognition, the learning objectives and also, the students’ motivation, whose research questions were formulated 
as follows: 

• What is the effectiveness of the hyperdocument developed about problem solving, in the scope of 
the achieved results and the developed cognitive processes? 

• What is the effectiveness of the scaffolding provided by the hyperdocument in the cognitive 
processes that occurred in problem solving tasks? 

A qualitative case study was designed involving eight students of the 6th grade. First, an interactive 
hyperdocument was built and validated with experts and users, based in the contributions of the self-regulation 
theoretical framework as well as problem based learning. The participants solved an initial problem before using the 
hyperdocument and another after the mentioned task. Data retrieved by students verbalizations in pre and post-test 
was subjected to content analysis and compared with a concept synthesis framework developed for that purpose. 
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Problem solving and self-regulated learning 

Problem solving involves the search for a path to achieve one or more objectives. During that process, 
students develop cognitive processes with different complexity degrees, activating, in that way, their metacognitive 
competences, constantly evaluating the work done. Relating the processes developed during problem solving, 
Rosario (2004) presents the self-regulated learning model PLEA (Planning, Executing, Evaluating) based in the 
work of Zimmerman (1997, 2000, 2002) (see Figure 1). 

The presented model is based in the operationalization of the problem solving process in three different 
phases: Planning, Execution and, at last. Task Evaluation. These occur in two cyclic logics (Rosario et al., 2007), 
since the process is in permanent updating, depending in the active part the student has in his own learning, in the 
sense that he needs to constantly monitor in order to evaluate and adjust the involved processes, reinforcing the 
inherent self-regulatory logic. 


PLANNING 



PLANNING PLANNING 

C "> C "i 


EVALUATION execution evaluation EXECUTION 



Figure 1- PLEE model of Self-Regulated Learning 

This process is then in a permanent dynamic update. This idea is defended by the literature in self- 
regulatory learning by suggesting that each student should leam a set of learning strategies that would allow him to 
take responsibility and control of his learning process (Rosario et al., 2007). 

Relatively to the presented model, Rosario and collaborators (2007) emphasize that “the logic inherent to 
the self-regulated learning process requires that each task should be planned, executed and evaluated” (p.23). In fact, 
according to many authors (Rosario 2004, Rosario et al., 2006, 2008), problems may present many “faces” from 
those that usually appear in Math classes to those of everyday life (e.g., calculate the best price for a product in the 
market, wear a green or yellow t-shirt, eat meat or fish, drive or walk, do the homework or copy it from a friend). In 
that sense, operationalization of the different phases of problem solving and the students getting used to them is of 
special value for their educational success, in the way that they stimulate the students to acquire a greater control on 
their learning, thus enhancing their motivation by the resulting sense of realization of the defined goals (Rosario, 
2004; Rosario et al., 2006, 2007, 2008). 

The developed study was meant to analyze those student developed processes and verify to which extent 
the developed hyperdocument has become an important influence in the qualitative and quantitative improvement. 
Thus, collecting verbalizations was of great importance, since, from its analysis, we hoped to obtain data to 
understand the cognitive processes developed by the students during problem solving and, also, the effectiveness of 
the developed hyperdocument. 


The hyperdocument 

We now proceed to the presentation of the interactive hyperdocument created for the empirical study using 
Microsoft Office PowerPoint 2007 tool, available in all computers with the Windows Vista and Windows 7 
operating systems. Taking advantage of the many functions available in this tool, the main objective was to provide 
dynamics and interactivity to the document, in order to make it a hyperdocument emulating the functioning of the 
human mind. 
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This interactive document intends to teach techniques of problem solving, through the operationalization of 
several stages identified in problem solving tasks (Jonassen, 2007). With this purpose in mind, we faced the need to 
develop an environment that allowed the student itself to develop its path, at his own pace, with full control over the 
process. With that purpose, we based the design in the principles of self-regulation developing an environment that 
allowed the student to decide what path to go, according to his motivation or pre-disposition. 

The importance of self-control on his own learning - verified, for instances in the choice of the path to go 
in the main menu of the hyperdocument (see Figure 2) - is an essential condition for self-regulation, and it can only 
be developed if the student is given the opportunity to do it autonomously (Rosario, 2004). 



Fig. 2- Main page of the Hyperdocument 


Aware of the role of motivation in self-regulation, it was our concern that, along the path through the 
document, an incentive was given to the student in order to maintain his motivational level not to give up, since the 
motivation takes an special role in the beginning of the development of self-regulatory processes (Rosario, 2004; 
Rosario et ah, 2006, 2007, 2008). 

The revealed concern about the active role of the student in the building of his schooling path translated in 
the planning of sequenced structures. That being so, first we tackle the need to solve problems, that won’t only 
appear in a schoolroom context; next, the operationalization of the solving process in stages and, finally, the 
presentation of a total of 4 problems, in order to apply the acquired knowledge. This sequence was elaborated in an 
open way, since it allows the student, at any given time, to modify his path, if he so desires. This possibility of 
building his own path inside the hyperdocument is on purpose allocated to the objective of putting the student in 
control of his learning. In consequence, this objective is accompanied of other, the need to implement educational 
practices that stimulate and develop the self-regulatory skills of students. 

His cyclic structure was conceived in a way to allow access to information through more than one path and, 
inclusive, to repeat that access (Sheiderman, 1992). 

The user goes through the many stages of problem solving. The operationalization of these stages was 
developed based in the book “(Des)venturas do Testas - Estudar o Estudar” (Rosario, 2004 ) and other references 
(Rosario et al., 2006, 2007, 2008). 

The problem selection had the objective of allowing the students (9-12 years old) to apply the various clues 
supplied by the application. In each problem, students train different techniques (e.g., table construction, rereading 
the problem, data comparison, attention to detail in the problem text), in a way to develop more efficient 
mechanisms of problem solving. The presented problems were adapted from problems found in school textbooks 
and the Internet, in an effort to match our specific needs and target-audience. 

In the end of each problem, the student was allowed to review the problem resolution phases again, analyze 
the problem again, and solve the problem with or without help. The inclusion of help along the path complies with 
the need to supply backup mechanisms (scaffolding images) to student learning, in a way to allow each student to 
build its path according to his level of knowledge. The scaffolding images appear as support pillars to learning, and 
are essential in any learning environment that promotes self-regulation. 

In this part of the presentation, we deemed needed the inclusion of help to problem solving in a way to 
supply scaffolding images, in the form of advice and tips to students, activating the learning done by the 
presentation. 

In case the student chooses to solve the problem by himself, it is always requested to confirm that option or, 
in case he so wishes, to go back and get help to find the answer. It is always mentioned to the student the need to 
analyse the problem again, independently of him solving it with or without help. 

In the end of the presentation, the student is confronted with a summary of the information provided along 
the document. The importance of method and persistence in the solving of problems is again remarked at this time. 
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Method 


The study was held in a Public School in the north of the country with eight students from the 6 th grade of a 
Science Class whose teacher was one of the authors. Taking into account that the students had to perform an 
exploration activity on the developed hypermedia application, with about 30-40 minutes duration, cooperative 
students’ selection was mandatory, so that it was possible to apply think aloud techniques. The investigator 
demanded the parents for authorization in participating in the study to perform, presenting its objective and 
framework. From the 24 students in the class, only half delivered the authorization that allowed them to participate 
in the study. 

Choosing of the eight students had into account their capacity to think aloud and verbalize in the three 
training sessions performed, and our intention to include in the sample students with different levels of school 
success. Thus, the sample was formed by 3 level 3 students, 3 level 4 students and 2 level 5 students, 4 boys and 4 
girls, randomly chosen, in each level, from the voluntary students. From the selected students, only 1 was 10 years 
old, being that the remaining 7 were 11. The average age was 10,86 years old. In order to ensure the confidentiality 
of the students, numbers were used to identify each investigation subject. 

Since we wanted to evaluate de effectiveness of the hyperdocument developed in the processes of problem 
solving, drawing a study architecture, based on 4 phases, was needed. In the first phase, the students performed 3 
training sessions, of about 15 minutes each, to improve their think aloud skills. These training sessions assumed 
great importance not only in the process of the participants’ selection but also to the improvement of their 
verbalization skills. This stage had the purpose of preparing the students to the think aloud task. 

To the second stage of the study, the students performed a pre-test in which they solved a problem, 
previously validated by teachers and students from the 5 th grade. During the resolution of this problem, the 
investigator recorded the students’ verbalizations, which were later transcribed. In this investigation we chose to do 
direct observation during the activities with minimal interaction with the students, solely clarifying doubts that came 
from the task given to them. 

In the third phase, students explored the developed hyperdocument and performed the task proposed 

within. 

In the last phase, the students were given a post-test, immediately after the exploration of the 
hyperdocument. The validation process was the same followed in the pre-test, so were the data gathering process 
and its transcribing. The operating mode in this phase was the same used in phase 2, with the objective of reducing 
the investigator intervention to a minimum. 

The data were recorded and then transcribed by the investigator. Gathered data were analyzed using content 
analysis techniques, in order to identify cognitive processes in the subjects verbalizations. Results were displayed in 
frequency tables. The categorization of the cognitive processes was based in the works of various authors in the 
scope of the influence of hyperdocuments and hypermedia applications in learning (e.g., Cromley et al., 2005; 
Azevedo et al., 2007a, 2007b; Greene & Azevedo, 2007; Moos & Azevedo, 2007). From this analysis we developed 
a summary table of the cognitive processes to identify in data (see Table 1) so as to more effectively analyze the 
transcripts of the student verbalizations that reveal the cognitive processes mobilized during the exploration of the 
hyperdocument. 
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Strategies 

Definition 

Self-Questioning 

Asks himself so as to help his learning process. 

Task Difficulty 

Mentions he feels difficulties in the resolution of the task, or that he 
does not understand the information. 

Correct inferences 

The student establishes a valid connection between matters through 
deduction or illation. 

Incorrect inferences 

The student establishes an invalid connection between matters through 
deduction or illation. 

Monitoring 

He comments on the adequacy of the achieved results or the adopted 
strategies.- 

Objectives 

Set of possible operations to execute or postpone, according to the 
different statuses we aim to achieve. 

Planning 

Coordinates available information, developing a specific way of action 
according to hierarchy of goals and sub-goals. 

Searching for data 
within the problem 

Searches for important data within the problem, aiming to 
develop/define a resolution strategy. 

Retelling of the 
problem 

Summarizes the problem in his own words, using the available data. 

Data registering 

The student registers the problem data. 

Rereading of the 
problem 

Reads the problem, or the problem part with the data or the question, 
again. 

Representation 

Uses drawings or schematics to help in learning. 

Reexamines the 

problem topics 

The student reexamines the situation, in order of evaluating the 
adequacy of his strategy. 

Summarize 

Summarizes what he read, interpreted or accomplished. 

Trial and error 

Resolution strategy. 


Table 1 - Summary of the cognitive processes 


Discussion 

Although the main objective of the study is to analyze the influence of exploring the hyperdocument in the 
cognitive processes developed by students in problem solving, the success obtained in the resolution of the proposed 
tasks is also worth of notice. In Table 2 we can compare the students’ success in the resolution of the proposed task 
in pre and post-test moments. 



Pre-test 

(#students / percentage) 

Post-test 

(#students/percentage) 

Correct answer 

3 / 37,5% 

6/75% 

Incorrect answer 

4 / 50% 

2 / 25% 

Gave up 

1 / 12,5% 

- 


Table 2 - Student results in pre and post-test moments 


Analyzing the table, we immediately realize the improvement in student results, by evaluation of the 
difference in the number of answers (correct and incorrect) given in the 2 evaluation moments. Although these data 
are not conclusive about the influence of the hyperdocument in the development of cognitive processes that 
potentiate the problem resolution, the increase in the percentage of student success in problem solving is, per se, 
worth of notice, and as such, an important argument in the analysis of the advantages of the hyperdocument. 

When analyzing the verbalizations of each student, it was necessary to develop an individual summary 
table where we counted the registry units collected from each student, that later originated a comparative general 
table. Table 3 synthesizes the information obtained in the analysis of the registries made during the pre and post-test 
phases of the 8 students under observation. 
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Processes 

Pre-Test 

Post-Test 


Example 

#/% 

Example 

#/% 

Retelling 
of the 
problem 

well, we have 3 
friends that want to 
cross to the other 
margin... hum... and 
the boat only carries 
130 kg” (Student 5); 

4/8,9% 

“we know Tiburcio has twice 
the cookies of Anacleto. . .” 
(Student 1); 

6/12,5% 

Represent 

ation 

“. . . let’s use drawings” 
(Student 1) 

2/4,4% 

The student schematizes the 
problem. (Student 7) 

1/2,1% 

Summariz 

e 

“... the boat that 
weighs 90 kg can go 
once, the 75kg boat 
once, and the 50 kg 
one, twice” (Studentl) 

1/2,2% 

— 

- 

Trial and 
error 

“Let’s see if 90 plus 
90... it’s 180, so it 
can’t be done like this” 
(Studentl); 

19/42,2% 

“let’s draw a random number” 
(Student 6); 

8/16,7% 

Searching 
for data 
within the 
problem 

— 

- 

“...we have to check the data 
we have already” 
(Student 1) 

1/2,1% 

Objective 

s 

“... if they weigh 50, 
75 e 90 kg I will have 
to know how many 
trips to cross” (Student 
2) 

1/2,2% 

“We have to know how many 
cookies Tiburcio has” (Student 
1) 

2/4,2% 

Rereading 
of the 
problem 

“ . . . it says here to 
calculate how many 
trips the boat has to 
make without sinking, 
and the boat won’t 
travel alone” (Student 
2) 

2/4,4% 

The student reads the part of 
the problem where Tiburcio 
notes his friend has no more 
cookies. 

(Student 7) 

12/25% 

Data 

registerin 

g 

.very well. I'll write 
the data. . .” (Student 4) 

3/6,7% 

“I will now take notes on the 
problem” (Student7) 

3/6,3% 

Planning 

“... I will read the 
problem again” 
(Student 6) 

3/6,7% 

“. . .1 will read the problem 
again” (Student 2) 

3/6,3% 

Correct 

inferences 

“it’s 215 kg, so they 
can’t cross the river” 
(Student 8) 

4/8,9% 

. .It leads me to believe that 
Tiburcio had 2 cookies” 
(Student 3) 

5/10,4% 

Incorrect 

inferences 

— 

- 

“that means Tiburcio will get 
twice the cookies of Anacleto” 
(Student 6) 

2/4,2% 

Task 

Difficulty 

“... this is difficult, I 
don’t see how I will 
solve this” (Student 5) 

3/6,7% 

— 

- 
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Reexamin 
es the 
problem 
topics 

“(adds 50 and 90 
again). . . this can't be 
it...” (Student 5) 

2/4,4% 

— 

- 

Self- 

Questioni 

ng 

“the 50 one and the 75 
one could go, but who 
would come back to get 
the 90 one, since it 
would pass the 
maximum boat 
capacity, so whose 
friend would come 
back to get the 90 kg 
one?” (Student 6) 

1/2,2% 

“. . .so, how many cookies 
Tiburcio has in his packet?” 
(Student 6) 

1/2,1% 

Monitorin 

g 

— 

“ 

“this number is not right. . .” 
(Student 6) 

2/4,2% 

TOTAL 

55 % 


88% 


Table 3 - Pre and Post-Test student verbalization examples 


Analyzing Table 3, the diminishing frequency of the “Trial and Error” strategy to less than half in the post- 
test activity is worthy of note. Also remarkable is that this strategy, in pre-test environment, was about 42,2% of 
analyzed processes, while in post-test it dropped to 16,7% of the total. This diminishing is interesting from a 
pedagogical point of view because it represents, in our opinion, a bigger awareness from students on the importance 
of developing a strategy to solve the problem. The need to develop plans to solve problems is mentioned several 
times in the hyperdocument, and is reinforced many times along the way. However, due to the fact that not all 
students performed all the proposed tasks (4 in total) of the hyperdocument and. as such, their path were shorter or 
faster than desired, this behavior is more evident in some students than others, having not been properly 
consolidated. 

As math teachers we observe, in our daily routines, in our students, a great incidence in the usage of the 
“Trial and Error” problem solving strategy. The daily used resolution methods reveal low concentration/attention 
and also the lack of an action plan. This way, one of the principles to adopt so as to promote success in these 
activities implies defining an action plan that allows the reaching of the right answer(s). Being so, it’s worth noticing 
the change in attitude in face of the problem, revealed by the diminishing frequency of the “Trial and Error” process, 
indicating awareness of the task solving process and a synonym of a need to define action strategies in the task 
resolution. The diminishing of the frequency of this strategy is, in educational terms, remarkable. 

Another behavior with significant changes is “Re-reading of the problem”. This behavior registered a 
significant improvement in its frequency, from 2 (about 4,4% of the total registered units) in the pre-test task, to 12 
(25% of the total) in the post-test. This increase is quite significant and we allocated in the learning performed by the 
students in exploring/using the hyperdocument. This behavior is several times recommended along the document, 
since the literature defends the need to develop self-regulating methodologies that potentiate metacognition and de 
resulting monitoring by the students, in the sense that they, themselves, develop a growing autonomy along the 
learning process (e.g., Azevedo, 2005; Azevedo et ah, 2005, 2007a, 2007b; Azevedo & Hadwin, 2005; Cromley et 
al., 2005; Donovan & Bransford, 2005; Rosario et ah, 2006, 2007, 2008). This autonomy grows thanks to the 
support (Scaffolding) received along the pedagogical process. In that sense, we consider the use of this 
hyperdocument an educational practice that stimulates the autonomy of the students. However, as Azevedo and his 
collaborators (2005, 2007b) proved, the use of an hyperdocument by itself is not enough for a significant improve in 
students results. For them to develop a greater depth of knowledge, the use of multiple representations of 
information is needed (e.g., notes, conceptual maps, summaries, tutorials). That is why different representations of 
information where adopted (e.g., text, pictures, tables), according to the limitations existing at the time, as the 
mastery in flash technology, that would give greater dynamic to the hyperdocument. The adaptability of the 
Scaffolds to the knowledge developed along the hyperdocument, also the self-regulatory training in class and human 
tutoring allow the aiding of students in developing more sophisticated mental models in the post- test results. 
Azevedo and collaborators (2005, 2007b) also propose that the students must analyze the situation, determinate the 
objectives define the strategies to achieve them and review their suitability and evaluate their learning by monitoring 
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the path and modifying strategies, objectives, plans and efforts according to the task conditions. That is why the 
need to reread the problem and defining objectives was so reinforced along the hyperdocument 

We highlighted the increase of the frequency in the processes “Reread of the problem” and “Retell of the 
problem” and the diminishing of “Trial and error”, synonyms of a superior analysis and consequent comprehension 
of the problem. These metacognitive skills are of great value when performing some tasks (e.g., problem solving) 
since they need the student to mobilize his previous learning in order to overcome new challenges. This adaptation 
to new situations is also highlighted by Donovan and Bransford (2005) when they pointed out the importance of the 
previous learning in this process and the importance of creating conditions that allow the consolidation of learning, 
associated with the variety of experiences with really important and significant facts to acquire new knowledge. 

We also noticed an increase in the “Retell of the problem” strategy frequency. This process was about 8,9% 
of the total registry units accounted for in the pre-test and increased to 12,5%. This is noticeable since, added with 
the above mentioned processes, it praises a superior awareness of the problem solving process. It’s a strategy that is 
valued in the hyperdocument, where we advise the student to retell the problem in his own words or with 
expressions from the text, in order to increase the level of understanding of the situation. 

Another behavior that could indicate a greater awareness of the task solving process is the “Summarizes” 
strategy. However, we noticed the extinction of this process in the post-test, even after it was used only once in pre- 
test. We believe that the fact the majority of students did not read the whole hyperdocument can explain this 
diminishing since, probably, they did not consolidate this resolution strategy. Another explanation is related with the 
concentration levels detected in students in the post-test. Taking into account it was performed immediately after the 
exploration of the hyperdocument, we noticed the majority of students was already somewhat out of focus and did 
not execute the task with the same commitment than in the pre-test. The students revealed less committed as the 
study went by. We think this fact deserves some attention from our part and we will get back to it in the final 
thoughts. 

“Searching for data in the problem”, “Defining objectives” and “Monitoring” had a negligible increase. We 
can observe a little increase, yet also of little significance, in the number of “Correct inferences” achieved. 

We can also point out the extinction of the “Task difficulty” expression. This behavior was 6,7% of the 
registry units in the pre-test and extinguished in the post-test. This behavior consists of the sensation of difficulty felt 
by the student when solving the task and is marked by statements like “. . .this is hard, I don't see how I can solve it” 
(Subject 5) or “...I can’t solve it” (Subject 6). 

This behavior was extinct in the post-test, indicating a greater control from the students in the difficulty of 
the given task. Another behavior that became extinct in the post test was “Reexamines the problem topics”. We 
believe this is related with a possible increase in the student self-confidence, which can also be negative. This 
behavior may also indicate a state of lower concentration/attention in which the student tries to answer the fastest 
possible just to finish the task. 

Unexplainably, the frequency of “Incorrect inferences” increased in the post-test. This fact appears to 
match what we said earlier about confidence, concentration and commitment of the students as the tasks unfolded, as 
we can analyze in the statement “if he has twice. . . and it says below that Anacleto has no cookies, so, twice none. . . 
is two. . .” (Subject 5). 

In this case, the lack of focus of the student is perceived, when he calculates the double of nothing. Also in 
the statement “(...) that means that Tiburcio is going to have twice the cookies that Anacleto has.” (Subject 6). In 
this statement, the student ignores the fact that one of the friends has no cookies already. These aspects may also be 
related with the task duration, indicating the loss of concentration during longer tasks. 

Relatively to the remaining used strategies, there were no significant changes. That may be related with the 
fact that some students interrupted their exploration of the hyperdocument too soon, which eventually hurt the 
acquisition of some competences (e.g.. Representation, Rereading of the problem. Attention to detail). Only one 
student got to the end of the hyperdocument, performing all proposed tasks. The remainder of the students 
abandoned the exploration after one or two tasks, thus not consolidating the information we aimed to transmit. Yet, 
they assimilated a part of the learning, like the need to reread the problem or not to try to find the answer by chance 
(trial and error). 

In total, there were no significant changes in the total number of accounted processes (45 in pre-test/48 in 
post-test). We can state, however, there was an increase in the quality of the cognitive processes used by students 
before and after the experience, materialized in a diminishing of the “Trial and error” of 58% and increases of 500% 
in “Rereading of the problem”, 100% in “Objective definition”, 50% in “Retelling of the problem” and 25% in 
“Correct inferences”. We also registered a diminishing of 50% in “Representation” and the appearing of “Incorrect 
inferences”. These events may be explained by some excess confidence revealed by the students, namely in the 
hyperdocument’s task resolution phase and the post-test phase, and also by a diminishing of focus that hurt the 
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unfolding of the post-test. The same reasons explain the diminishing of 50% in “Representation”. The excess 
confidence was notorious in some students when they read the problem too fast, without paying attention to some 
details as was the case in the post-test, neglecting the fact that one of the students already had no cookies or reading 
only the initial information that the package of one of the friends had twice the cookies. This lack of attention to 
detail increases with the speedy reading of the problems and by their apparent ease, which contributes to explain the 
bad results obtained in this dimension 

Relating “Self-Questioning”, the registered frequency in pre and post-test was the same and, curiously, 
from the same student. It is a student with a very critical mind but, however, too much confident, which hurt them in 
the solving of tasks as the one on post-test, where he neglected the already mentioned fact that one of the friends 
already had no cookies. 

The remaining behaviors had small fluctuations, of little relevance percentage. 

The global positive results registered allow us to seriously consider the use of this type of cognitive tools in 
knowledge acquisition. As we previously stated, this use is not enough alone. It must be accompanied with a 
meditation about its purpose and components (Jonassen, 2007; Ramos, 2007; Silva & Silva, 2005), so as to allow the 
students to develop adaptation and work monitoring skills. Obviously, this path is quite long, and it belongs to the 
teacher to promote student autonomy, allocating Scaffolds of various types and grade to the learning, so that the 
student is confident in his path. 


Conclusion 

Considering the first research question - What is the effectiveness of the hyperdocument developed about 
problem solving, in the scope of the achieved results and the developed cognitive processes? - the obtained results 
allow us to say that the hyperdocument positively influenced the participating students. This influence may be 
observed: i) in the increase of correct answers in post-test, ii) in the significant diminishing of the frequency of 
“Trial and error” processes, iii) in the significant increase of “Rereading of the problem”, iv) in the slight increase of 
behaviors of significance to the success in the tasks, as “Rereading of the problem”, “Retelling of the problem”, 
“Monitoring”, “Objective definition”, “Correct inferences”, v) in the diminishing of “Task difficulty” and vi) in the 
enthusiasm of the students in the exploration of the hyperdocument. Although some more significant than others, 
these differences suggest advantages in the use of the hyperdocument in the building of new learning. Its 
architecture, according to the metacognitive and self-regulatory objective of increasing the autonomy and sense of 
adaptation of the students, is adequate to its use by students, individually or, eventually, in pairs. However, we need 
to point out the need to accompany the use of the technology with a meditation on the educational practices that 
would support it (Jonassen, 2007; Ramos, 2007; Silva & Silva, 2005), something that goes along the statement of 
Azevedo and cooperators [5,6] when they pointed out the difficulty the students feel on handling monitored learning 
in hypermedia environments. We conclude then that technology alone is of no use, if it is not accompanied with a 
daily pedagogical basis, supported by the discourse and teaching practices of the teachers. 

Considering the second research question, we aimed to know if the scaffolding provided to the students along the 
hyperdocument were effective in the acquisition of new learning and, as such, capable of influence the cognitive 
processes developed during the performing of themselves. These Scaffolding appear thru the hyperdocument in the 
form of suggestions or advice in the solving of problems, in a way the students may develop process based 
knowledge about problem solving, through common performance. On more elaborate hyperdocuments or on other 
knowledge areas, the use of process based Scaffolding may be essential in acquiring that knowledge (Azevedo, 
2005; Azevedo et ah, 2005, 2007a, 2007b; Azevedo & Hadwin, 2005; Cromley et ah, 2005; Azevedo & Jacobson, 
2007; Greene & Azevedo, 2007). 

This study revealed some lack of prepare of the students in the performing of exploratory type tasks where they 
are given great control over the situation. This fact may be made evident by the hurry of the students in finishing the 
path in the hyperdocument. That is why we recommend the possibility of “jump to the end” that is given to the 
student be removed from the initial stages. This realizing goes along the statements of Azevedo and coworkers 
(2007b), when they mentioned the learning in hypermedia environments raises greater accountability in students, 
thus feeling a greater difficulty in adopting strategies to build new knowledge. The authors also point out that, in 
general, the students use ineffective learning strategies, without defined objectives. This reinforces the need to the 
educational system to encourage the teaching of self-regulation strategies in the classroom. Teachers must provide 
their students with acting mechanisms in these kind of environments, so that they develop knowledge in a more 
autonomous, confident and effective way. 
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The use of the hyperdocument by the students appears to be beneficial to the greater awareness of the students 
about the importance in establishing ways of acting in solving problems. It also creates a greater involvement from 
their part in their learning, as it permits that they develop an autonomous work, according to their previous learning 
and/or knowledge. Along with the developed hyperdocument itself, the use of technological means like the 
computer or the interactive blackboard allows for greater involvement and predisposition towards learning by the 
students, developing intellectual partnerships in building knowledge and meditating about their learning (Azevedo & 
Jacobson, 2007). 
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Abstract 

Autism, a spectrum of neurological disorders, persists over the lifetime of an individual. It affects 1 in 88 
children. Individuals with Autism Spectrum Disorders (ASDs) have problems with communication, behavior 
(restrictive and repetitive), and social interaction. In the past, ASDs were poorly understood and not well publicized, 
but now, public service announcements, blogs, magazines, newsgroups, radio programs, a host of websites, and 
journal articles on ASDs are prevalent. As a result, many families are aware of the signs and the types of 
interventions that are available. Parents face a number of significant challenges and stressors as they seek 
inventions, plan for the future, obtain resources, balance the needs of all children in the family, and work through 
each day’s events. Deficits in social skills, communication, and behavior are found in varying degrees in individuals 
with ASDs. Co-morbid conditions, lack of Theory of Mind, and varying levels of intelligence are present as well; 
families use a variety of technology and non-technology interventions to mitigate deficits. This work explores some 
of the challenges of parents (things parents wish others knew about their life), the technologies parents select, their 
concerns about technology, and demographic data. 


Introduction 

Several pervasive developmental disabilities are included in the Autism Spectrum: Autistic Disorder, 
Pervasive Developmental Disorder - Not Otherwise Specified, and Asperger Syndrome (Center for Disease Control 
and Prevention, 2012). Children with ASDs may have additional challenges: depression, conduct disorders, sensory 
integration dysfunction, feeding issues, learning disabilities, anxiety disorders, ADD/ADHD, repetitive movements 
(stimming), obsessive compulsive behaviors, motor skill difficulties, seizure disorders/epilepsy, impairments in 
adaptive behavior, an aberrant regulation of emotion, discomfort with eye contact, late onset of speech, difficulty 
expressing empathy, as well as problems with communication, social interaction, and behavior (Loeber & Keenan, 
1994; Madsen, Kaliouby, Goodwin, & Picard 2008; Myles & Simpson, 2002; Portway & Johnson, 2005; Reiss, 
2009). The combination of any of these is a recipe for parental stress and anxiety, and many studies on parents of 
children with ASDs report stress, higher levels of stress than parents of children with other disabilities, persistent 
stress throughout the child’s lifetime, increased depression, restrictions of roles and activities, marital stress, and 
diminished physical health (Fleischmann, 2004; Ingersoll & Hambrick, 2011; Myers, Mackintosh & Goin-Kochel, 
2009; Pisula, 2007; Stoner, Bock, Thompson, Angell, Heyl & Crowley, 2005; Shu 2009). 

One promising tool for enriching the lives of families struggling with ASDs and co-morbid conditions is 
technology. Broadly defined as any electromechanical device which helps users accomplish work, engage in leisure 
pursuits (entertainment), learn new content (education), or both (edutainment), some technologies used with learners 
diagnosed with ASDs include: video-modeling, iPads, iPod Touch, laptop computers with Internet Access, DVD 
Players, virtual environments, augmentative and alternative communication devices, CAI, PDAs, e-Books, Nintendo 
DS, Wii, and a host of other devices and tools. Many parents of children diagnosed with ASDs are using technology 
to help their children gain academic, listening, and social skills. In addition, they are using technology to manage 
their child’s time, remind the child of critical daily activities, help the child develop leisure pursuits and build 
communication skills. Understanding the needs of families can create a platform for inclusion, acceptance, and 
technology integration for teachers, administrators, instructional designers, and therapists. 
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Purpose 


The purpose of this exploratory investigation was to acquire insight about parents’ challenges with an 
Autistic child, examine demographics, and identify technologies used. In order to gain experience for further 
inquiry, the following questions were asked: 

1 . ) What do families wish others knew about their life with a child in the Autism Spectrum? 

2. ) What technologies are used by families to address the problems faced by their child with an ASD? 

Theoretical Foundation 

This inquiry is situated in Adult Learning Theory - specifically experiential learning based on the work of 
Dewey, Piaget, Kolb, and Lewin. Adult learners in this study are parents of children with ASDs, and they learn to 
understand their child and discover the child's unique needs and interests by direct interaction with the child. The 
parents learn to work with their child by doing so, and they reflect on the strategies that worked as positive 
reinforcement, the methods of discipline that were effective, and the educational experiences that were most 
successful. Parents’ gain knowledge of the therapies and instructional strategies that work and those that do not by 
observing their child, trial-and-error, the child's behavior, and their own internal gage of the amount of progress they 
witness in their child. The parents’ learning is based on their personal and environmental experiences, and it requires 
active involvement, problem-solving, analytical ability, and memory. No teacher is present; the parent learns from 
direct experience rather than lecture, conversation, or readings. 

Methodology 

Participants 

Parents in Texas who have children diagnosed with Autism Spectrum Disorders (ASDs) were invited to 
complete an online survey in an attempt to determine parental challenges and the technologies used. In order to gain 
information, a survey was created, reviewed by statisticians, pilot tested, and placed online. After gaining IRB 
approval, the link to the survey was sent to electronic newsgroups whose focus was ASDs. The survey had four 
sections: Background Information, Demographics, Physical and Behavioral Characteristics of the Child with ASDs, 
and Educational Options. Survey responses were gathered from 2008-2010 using a snowball sample technique 
which began from links posted on three known sources: online discussion groups that focused on ASDs, the 
Interactive Autism Network which publicized the study on its list of Autism Studies in Texas, and the TARA 
Website. In addition, families were recruited through face-to-face meetings at local Autism Societies and through 
word of mouth. 

Coding Responses 

The parents’ responses were reviewed for themes using content analysis (Miles & Huberman, 1994). The 
primary coder made a list of tentative themes, and two additional coders reviewed responses. As a result, the original 
themes were revised, combined, and expanded until there was agreement. Once the themes completely captured all 
the concepts in the data, parents’ responses were inductively reviewed again; sorted into primary and secondary 
themes, and the number of statements reflecting a primary and secondary theme were clustered and tallied. As an 
example, one primary theme was training, and it included parents’ references to an awareness of ASDs, family 
knowledge of ASDs, teacher and parent training. 

Findings: Challenges of Parents: (Things They Wish Others Knew) 

A total of 307 individuals completed surveys. Those were filtered to include only those respondents who 
indicated that they resided in Texas. Demographic data from those surveys along with the open-ended answer to the 
question, “What do you wish others knew about your life with a child in the Autism Spectrum?” were analyzed 
using SurveyMonkey and Excel. Exactly 220 responses were reviewed; 195 open-ended answers were coded by 
multiple reviewers and those presented an interesting story. The shortest response was five words: “Just how 
difficult it is.” The longest response was about 1.125 pages, and it contained 1,210 words. The coders came to 
agreement on 19 themes: need for family support, hard/difficult, unfair judgments, reasons for tantrums, kids with 
ASDs need patience, kids with ASDs need understanding, cost, worry, isolation, exhaustion, stress, frustration, need 
for structure, endless issues, insurance, training, positive descriptors, negative descriptors, and N/A (not applicable). 
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A majority of respondents mentioned at least one theme in their answer (53.3%). Two to five themes were 
mentioned by 30.8% of the respondents, and 15.9% made comments that could not be placed into any of the 
identified themes. An example of the latter: “I have noticed quite a few parents (in my area) who must think it’s ‘in’ 
to have a child who is autistic and have diagnosed their children on the Internet. They have completely turned me off 
from attending parent classes in my local area.” Many studies on parents of children with ASDs identify parental 
stress, heightened risk of mental health problems, chronic health conditions, a lower quality of life, difficulty 
obtaining services, inadequate service delivery, lack of collaboration, increased depression, anxiety, and other 
negatives as themes that emerge from parental perspectives on life with a child with an ASD (Fleischmann, 2004; 
Stoner, Bock, Thompson, Angell, Heyl, Crowley, 2005; Pisula, 2007; Shu, 2009; Ingersoll & Hambrick, 2011). The 
six most commonly reported themes are listed below. 



1.) Parenting a child or children with an ASD H very hard/difficult. 


2. ) Parents used positive adjectives to describe their child and what they have 
learned. 

3. ) Other individuals watching their child should not judge the child or his 
parents. 

4. ) Negative descriptors for life surfaced. 

5. ) Training is needed. 

6. ) Stress is present. 


Theme #1: Parenting a Child or Children with an ASD is Very Hard/Difficult 

Many factors contribute to a difficult life with a child (or children) diagnosed with ASDs. Some factors are 
tantrums or meltdowns on the part of the child, self-inflicted injury, over stimulation, sleep deprivation, and flight or 
fight (Fleischmann, 2004). Often, an unmet physiological, biological, or emotional need that cannot be verbalized 
causes the child to become frustrated and have a meltdown. Parents straggle with meltdowns, therapy selection, 
education, feeding problems, and other issues (Ivey, 2004; Starr, Foy, Cramer & Singh, 2006). 

An individual with an ASD may pick his skin severely or inflict other forms of self-injury by punching, 
hitting, or biting. These are reactions to anger, stress, or frustration, and they can create stress on parents. Many 
individuals with ASDs have either an over-active (hypersensitive) or under-active ( hyposensitive) sensory system 
that causes them to react to pain, sound, and noise differently than their neurotypical peers. In a classroom setting, 
an individual with an ASD may simultaneously hear lights humming, coughing, others talking, the A/C, the pencil 
sharpener, and the teacher talking and have difficulty filtering these sounds. Difficulty filtering and adjusting to 
noise and other stimuli may make it uncomfortable to be in certain settings - florescent lighting, loud noises and 
crowds can make a setting painful and overwhelming. These problems necessitate extra planning, avoidance of some 
settings, and Occupational Therapy determine and address sensory integration dysfunction and other issues. Normal 
appliances can have a similar negative effect - noise from a microwave, dryer, or vacuum may cause a person with 
an ASD to become fearful and react by covering his ears, running away to escape the sound, or screaming. 

Problems getting an adequate amount of rest have been reported in children with ASDs, and these problems 
are more severe and more frequent than those found in typically developing children (Cotton & Richdale, 2006; 
Hoffman, Sweeney, Lopez- Wagner, Hodge, Nam & Botts 2006; William, Sears & Allard 2004). If children with an 
ASD are not rested, they are more susceptible to frustration and a loss of control. When children do not sleep well, 
their parents do not either. This further complicates life, and all of the above coupled with a lack of support and 
respite, little understanding from family and friends, isolation, and worry, make the life of a parent extremely hard 
and difficult. Some of the parents expressed these sentiments: 

“It’s hard! He looks almost fine since he is high functioning and cute. When he acts up, people are inclined to think 
it’s a behavior problem and not ASD overload.” 

“HOW HARD IT IS TO DO ‘NORMAL’ ACTIVITIES. EVERYTHING IS A STRUGGLE.” (This response was 
typed in capital letters.) 

“I wish others knew how hard it is to receive no extra help.” 

“It is a very challenging and difficult experience. And it is especially difficult to see that your child desperately 
needs your help to learn and develop.” 
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“It is hard but we try to make things as normal as possible, so don't look at him like he is strange when he waves 
and asks your name. . . just smile and wave back.” 

Parents’ lives are also difficult because their child has varying degrees of deficit in three areas: 
communication/language, behavior (restrictive and repetitive), and social skills. The child with an ASD may be non- 
verbal or have problems with expressive or receptive language or pragmatics (practical use of language). The child 
may also exhibit delayed echolalia (repetitive use of words or phrases from television programs or stories) as a 
substitute when he knows he should respond, but does not know what to say. Delayed echolalia is also used to make 
a request or protest an action. Communication problems make it difficult for the child to get his/her wants and needs 
met by making appropriate demands within the environment. The frustration resulting from language deficits and 
the need for professional assistance adds to the challenge of parents. 

The child with ASDs may engage in restrictive or repetitive behaviors. They may have a special area of 
interest, leant everything about that area (trains. Old West, History, Mine Craft, SimCity, or other), and constantly 
repeat things about that narrow area. This makes communication with others difficult, because constant and 
repetitive conversation about the interest is unwelcome. The individual with an ASD may not understand that others 
have different interests and passions (Theory of Mind - Baron-Cohen 1995; Baron-Cohen 2008), and do not want to 
hear about a single interest repeatedly. Attempts by others to display boredom and disinterest are not perceived very 
well by the individual with an ASD. Quite often, they fail to effectively read body language or facial information 
describing another person’s level of interest. Repetitive behaviors are known as self-stimulatory behaviors (shaking, 
noises, flapping, peeling paint from the walls), and they are used for calming or reacting to boredom or stress. 
Sometimes self-stimulatory behaviors appeal' strange, and the individual that engages in these behaviors may be 
misunderstood, isolated or ridiculed by his neurotypical peers. 

Social skills are another area of deficit prevalent in individuals with ASDs; direct instruction, modeling, and 
multi-level support are needed to address this shortcoming. Same aged peers can provide the role modeling 
necessary for demonstrating and shaping appropriate behavior, and that can build social acceptance and foster 
collaboration. Supportive teachers that understand the disorder, are knowledgeable, and supply consistent positive 
reinforcement and patience can nurture social skill development, encourage peer-to-peer collaboration, and reinforce 
appropriate behaviors. Care should be taken to make sure all personnel (instructors, paraprofessionals, and aides) are 
well-trained and experienced, because untrained paraprofessionals can impede social skill development and 
acceptance and create confusion and overdependence (Zager & Shamow, 2005). Family support is critical for 
emotional balance, academic achievement, communication, physical needs, and general development and progress. 
Multi-level support systems can help individuals with ASDs function and learn appropriate behaviors and social 
skills. Support systems neutralize the isolation, anxiety, and depression of those with ASDs. (Brewin & Renwick, 
2008; Marshall, 2002; Muskat, 2005; & Rayner 2005). 

Theme #2: Parents' Positive Adjectives Describing Their Child and Their Life 

Parents used positive adjectives and phrases to describe their child. The literature is full of negatives in the 
lives of parents, but the second most populated theme contained positive messages where parents have accepted 
their child, love the child, regard his/her care as a “blessing,” and feel that the child is a “gift.” The most common 
descriptors in this theme were “loving,” “sweet,” and “beautiful, gift, and wonderful child." Hastings, Kovshoff, 
Ward, Espinosa, Brown & Remington (2005) report the measurement and analysis of positive perceptions of parents 
of children with ASDs. In addition to stress, both fathers and mothers identified positive perceptions about their 
child and his impact on themselves and other family members. Some of the comments: 

“We are not “sorry” he is the way he is- he is a bright, loving child who is the joy of our lives’ he simply has some 
challenges, as do we all.” 

“Frightfully demanding, but it evokes a love you will never know.” 

“I am just trying to accept my child for who he is and help him grow. I love him.” 

“While it is the most difficult thing my husband and I have had to go through, it is also very rewarding. These 
children are so loving and giving. They are extremely smart.” 

“He is lovable and has so much to offer if give [en] the chance and the oddities were acceptable to all children.” 
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Theme #3: Stop Judgmental Observations 

Parents reported that they were doing their best, and they reported resentment to the comments and stares 
they receive from family members, friends, and onlookers when they are out in public and their child misbehaves. 
Parents reported that they do not appreciate the stares and judgments of others who react to them as though they are 
bad parents, lack the ability to administer discipline, or permissively deal with their child. Some quotes from parents 
are included below: 

“For the public to stop judging us and labeling our kids as bad kids or poor parenting. It is a major challenge raising 
kids on the spectrum and we do the best we can and especially when there isn’t any other form of support for us. 
Instead of judging us, try asking if we need help.” 

“I wish others would try and see him as 1 do- an individual with gifts and strengths who can be loving and kind if 
given an opportunity. If they would attempt to communicate rather than judge they would find a person worth the 
work of getting to know!” 

“A day in the life; don’t judge when you see me handling tantrums, etc. in public. You have no idea what we have to 
do to manage the situation.” 

“Do not judge others if you think there is nothing wrong with their child. Our daughter is high functioning, but very 
tall and big for her age. Almost 7 but looks like 10. It is a double blow as she acts younger than her age, but looks 
older than her age.” 

“That they walk in our shoes before judging.” 

Theme #4: Negative Adjectives for Life with a Child with ASDs 

Several parents used negative adjectives to describe their life with a child diagnosed with an ASD. The 
most frequently used adjectives were lack, sad, and depressing and demanding. Lack was expressed in terms of 
resources, available professionals to provide assistance, and support from families, friends, the school system, and 
insurance providers. Reflections reported include the following: 

“That being a single mom with 1 Autistic Child and another with traits is difficult and that we don't have many 
resources here in our town.” 

“I guess I wish that people understand how sad it is to grieve your child on sometimes a daily basis. Having a child 
with special needs is grueling on a marriage, which is fairly common knowledge.” 

“The toll it takes on a family, the fights we have with insurance to get coverage for services, the therapy that is 
needed to cure it.” 

“You’re given this diagnosis and pretty much sent on your way to figure it out for your child. The school district 
gives the minimal amount of therapy so you must figure out the rest on your own.” 

Theme #5: Training is Needed 

Parents expressed the need for training on ASDs and supportive services at many levels: child, parent, 
family, community, and teacher/school personnel. Both private (immediate and extended family) and public spheres 
(community, businesses, and school systems) of life are reflected in the different types of support needed. According 
to the respondents, some of their family members and friends need to be trained on the characteristics and behaviors 
of children with an ASD. Community members and business personnel need training on the importance of inclusion 
for individuals with ASDs, teachers need effective strategies for working with children and adults with ASDs, and 
parents need training on behavior and social skill development. Some of the comments that reflected the need for 
training are listed as follows: 

“What goes on at home. To my friends we seem like a normal family. Some think E. is just spoiled, they don't 
understand what a meltdown is like for her and for us.” 

“That he isn't retarded; he has at least average intelligence, but it is a matter of teaching him in a way that he can 
learn.” 

“He wants to have friends, yet other families don’t teach their children to be good mentors to these children." 

“That just because an individual has a diagnosis of Asperger Syndrome does NOT necessarily mean they are 
“higher” functioning than an individual with a diagnosis of PDD-NOS or autism. . .their symptoms are instead 
manifested differently.” 
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“I would [like for] people to understand what it is like raising a child with autism and we need more services 
through the school district support with the behavior, and social training for the child and parents.” 

“That he is not mentally retarded, and that he has normal intelligence. That he won't GROW OUT OF IT, 

Grandma!” 

Theme #6: Stress is Present 

The stress experienced by parents of children with ASDs persists throughout the child's lifetime and 
manifests itself in an increased risk of depression, restrictions of roles and activities, marital stress, and diminished 
physical health (Fleischmann, 2004; Ingersoll & Hambrick, 2011; Myers, Mackintosh & Goin-Kochel, 2009; Pisula, 
2007; Stoner, Bock, Thompson, Angell, Heyl & Crowley, 2005). Stress was the sixth theme into which the highest 
number of respondents’ comments fell. Some reasons cited for stress were marital, financial, psychological, and 
emotional pressure. Stress results from the child's behavior, level of functioning, educational needs, interventions, 
parental expectations, worry about the child's future, and the need to make provisions for the child after the demise 
of the parents. Parents expressed doubts about the safety of their child with an ASD, and they had a tendency to be 
overprotective. As a result, they tended to limit their child's independent skill development (Ivey, 2004). Many 
parents feel isolated, and the constant demand of finding and financing programs to help their child leaves many 
with little time to recharge themselves and few outlets for relaxation and enjoyment (Fleischmann, 2004). Several 
comments were indicative of this: 

“That it’s harder than it looks, that even if he looks normal at first glance, there are so many aspects to it once you 
get to know him. It’s very draining financial wise and that it has an effect on my marriage, and not always a good 
one either.” 

“The feeling of isolation from the rest of the world and the worries associated with raising a child in this day and age 
increase three fold making stress unbelievably high.” 

“How stressful it is not having personal and/or school/state/federal resources/assistance; how psychologically 
oppressive it is to feel that our child’s future depends SOLELY on our research, our interventions and our decisions 
- there is really no one professional to turn to for information and guidance.” 

“The constant worry about time running out to recover your child - am I selecting the right therapies? The right 
supplements? What have I missed that could be the key to help my child? It is a low-level chronic stress that never 
goes away.” 

“Stress is never ending. Just when you move through one problem or crisis, another one pops up. We need people to 
help with everyday care issues. One size fits all programs do not work. We need help early on when they are young 
so that they maybe won't need it later.” 

Nineteen themes emerged from the responses (195). The number of themes suggests that life for parents is 
complex and multifaceted and necessitates simultaneous attention to many issues. The six most frequently identified 
themes were discussed, but other themes included worry, the endless obstacles, and the need for understanding and 
compassion for children in the spectrum. Many parents reported using technology to address some of the issues and 
challenges they face. 


Technologies Parents Select 

Technology tools have the benefit of being able to individually address different needs and different levels 
of functional capability. They can provide flexible presentations, repetition, individualization, motivation, locus of 
control, prompting, fading, visual lessons and modeling, and record keeping. Parents of children diagnosed with an 
ASD are using technology to help their children gain academic, listening, social, time management, leisure, and 
communication skills. Researchers have discovered that computer use with learners diagnosed with ASDs increases 
competence in speech (receptive language), and other areas (social, behavioral, and cognitive), improves attention, 
motivates the learner, and reduces problem behaviors (Cramer, Hirano, Tentori, Yegnayan & Hayes, 2011; Dettmer, 
Simpson, Myles, Ganz, 2000; D'Ateno, Magiapanello & Taylor, 2003; Escobedo, Nguyen, Boyd. Hirano, Rangel, 
Garcia-Rosas, Tentori & Hayes, 2012; Gentry, Wallace, Kvarfordt & Lynch, 2010; Hourcade, Bullock-Rest & 
Hansen, 2012; Hetzroni & Tannous, 2004; Kagohara, 2010; MacDonald. Clark, Garrigan & Vangala, 2005; 
Mechling, Gast & Seid, 2009; Moore & Calvert, 2000; Muskat 2005; Myles, Ferguson & Hagiwara 2007; Tartaro & 
Cassell, 2007). Parents across Texas reported using the following technologies to help their children gain functional, 
language, and academic skills: video games, assistive technologies, educational software (CAI/CAL- Ed Mark's 
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Reading Program, ComfyLand Software), listening programs (Therapeutic Listening, Samonas Sound Therapy, 
Auditory Integration Therapy), and Neurofeedback. 

Technology Concerns 

Technology can offer many benefits to both children in the Autism Spectrum and their parents. However, it 
cannot be considered a panacea to mitigate all skill deficits. While it has tremendous positive potential, it should be 
used with restriction. This is important for all children, especially those with ASDs. They may become obsessive 
about the hardware and software they use often, and they may model behaviors and actions they see and use 
language they hear in games and programs. This can have devastating consequences when real life clashes with the 
animated and surreal presentations the children see. In addition, they can become so comfortable in created reality 
(Lego StarWars, Super Mario Brothers, Pokemon) that they flee face-to-face encounters. While the games in and of 
themselves can be very entertaining, the constant use of them can keep the child from learning other important 
lessons and social skills. A balanced approach is advocated at all times; technology should provide both 
entertainment and academic skill development. Technology is highly visual and motivating; it should be 
incorporated into academic work to balance the “boredom factor.” Reading assignments on the iPad and visually 
presented worksheets can be very motivating and help the child persist through academic work. Screening programs 
from the Internet and game consoles (Wii, PS3, Nintendo DS, DVD, etc...) is important for all children, especially 
those with an ASD. Rating systems and monitoring software should be used regularly to make sure children with 
ASDs are not exposed to inappropriate, crude or offensive content. Children diagnosed with ASDs may not 
understand the language and motives of others (Little, 2003), so it is particularly important that parents and teachers 
provide extra control and monitoring to make sure they are protected from cyberbullies, pedophiles, and negative 
influences on and offline. 


Demographics of Survey Participants 

Between 2008 and 2010 respondents generously shared their technology choices, needs and issues. Only 
respondents who indicated that they resided in Texas were included in this analysis (220 responses). All of these 
responses were from mothers of children with ASDs (100%), and the diagnoses their children received most 
frequently were Autistic (45.0%), Asperger Syndrome (25.0%), and PDD-NOS (30.0%). Most of the time, a team of 
professionals performed the diagnosis, followed by a Clinical Psychologist, a Psychiatrist, and lastly a 
Developmental Pediatrician or School Counselor/School Psychologist. The majority of children were male (85.8%); 
ASDs affect more males than females (Center for Disease Control and Prevention, 2012). Most mothers (90.7%) 
reported that their child with an ASD was physically healthy. When asked about the race of their child, most mothers 
reported that their child was Caucasian (77.7%), Hispanic (15.0%), and African American, Asian, or Native 
American (7.3%). The most frequently reported age range of the mothers was 35-39 (30.1%); this was followed by 
40-44 (21.8%), and 45-49 (17.1%). About 57.3% of the mothers indicated that they were employed outside their 
home in the following careers: Teaching, Nursing, Administrative Assistant/Secretarial, Sales, and Supervisory or 
Management positions. Many non- traditional job titles were reported: President of a software company. Usability 
Website Designer, Pediatrician, Radiation Therapist, Chemical Processing Technician, CPA, CEO, and others. 
Household income rounded to the nearest dollar: $50,000-59,999 (11.1%), $40,000 - $49,999 (10.1%), $100,000- 
$109,999 (8.7%), $80,000 - $89,999 (8.2%) and $170,000 - $179,999 (8.2%). Insurance coverage for some of the 
child’s interventions (58.7%), and an annual estimate of $1,000 - $9,999 (42.8%) was reported for “out-of-pocket” 
intervention expenses. 


Conclusion 

Children with ASDs have a lot to offer the world. They need multi-level support systems that are flexible 
and evolve as they grow and change and technology to help with academic, social, and behavioral challenges. In 
addition, parents of children with ASDs face many obstacles; they need effective technology tools, strong systems of 
inter-family and intra-family support, understanding, and training to help them help their child. This study provided 
insight on the parental perspectives and technology use of families in Texas who are living with and loving a child in 
the Autism Spectrum. It sets the stage for further research on the stress of families, their needs, and the implications 
of technology. 
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Abstract 

Children in the Autism Spectrum (ASD) suffer from a pervasive developmental disorder which renders 
them unable to communicate effectively, behave appropriately and engage in meaningful social interactions with 
others. It is difficult for children with an ASD to understand their needs and feelings and those of others. Quite 
often, they are misunderstood in social settings, and they require explicit instruction to help them “navigate” the 
social world. Their inability to communicate effectively makes their life frustrating, and as a result, they tantrum, 
display aggression, and behave inappropriately. In an attempt to ameliorate their social skills and communication 
deficits, many researchers, therapists, and parents select technology tools to help children with an ASD gain 
functional skills, learn appropriate behaviors, acquire academic skills, and learn to express themselves. This article 
explores promising applications of technology and presents preliminary results from an exploratory study designed 
to discover the technology preferences of five children with ASD between the ages of 8 and 12. 

Introduction 

Autism Spectrum Disorders (ASDs) is a term which refers to several pervasive developmental disabilities: 
Autistic Disorder, Pervasive Developmental Disorder - Not Otherwise Specified, Asperger Syndrome (Center for 
Disease Control and Prevention, 2012). Deficits in communication, social interaction, and behavior are hallmarks of 
ASDs, and each individual in the spectrum has varying degrees of impairment in each of these areas. The 
programming that works for one individual does not work for all individuals in the spectrum, but technology tools 
have the benefit of being able to individually address different needs and different levels of functional capability. 

This makes technology suitable for individuals in the spectrum. All of the following are benefits of technology: 
broad applicability to many areas, individualization, repetition, leveling, motivation, prompting and fading, visual 
appeal, precisely defined functions, record keeping, and locus of control. The next section of this paper describes 
current research on technology that shows promise: PDAs, MOSOCO, an Embedded Social Network, Interactive 
Visual Supports, and Multitouch Tablets. These projects apply the beneficial aspects of technology tools to areas of 
deficit found in individuals with ASDs: social skills, language/communication, and behavior. The final section of 
this paper describes some observations of technology preferences from an exploratory study of 8-12 year-old 
children with an ASD. Children’s technology-using behaviors can help us learn about their preferences, needs, level 
of understanding, motivation, information processing, and view of the world (Hourcade, Bullock-Rest, & Hansen, 
2012). 


Technology Use in Autism Spectrum Disorders 


PDAs 

Personal Digital Assistants (PDAs) can help students with ASDs who have difficulty with tasks involving 
executive functions: planning, goal setting, memory, and shifting between visual and auditory stimuli while 
attending to important features of a task (Gentry, Wallace, Kvarfordt, & Lynch, 2010; Mechling, Gast, & Seid, 2009; 
Myles, Ferguson, & Hagiwara, 2007). Many children in the spectrum have difficulty maintaining attention and 
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recalling information when they engage in tasks that require many steps. Though they have the intelligence needed 
to perform well, they tend to perform poorly at home and school, because of their inability to independently 
complete tasks that are scheduled. They may complete homework and assigned tasks, but forget to turn the work in, 
or they may have a planner and forget to reference the planner or they may forget to add an item. PDAs can be 
customized to include visual (video, pictures) and auditory prompts, structure, recording capability, cameras, 
calendar features, Internet connectivity, GPS, and organizational support (To Do Lists, and Programmed Reminders) 
which increase independence, decrease a reliance on external prompts, and assist with time management. 

Gentry, Wallace, Kvarfordt, & Lynch, 2010 found that 91% of high school students with ASDs who had 
executive dysfunction learned to operate PDAs (Palm ® Zire 31) and use them as an assistive technology tool to 
improve their performance on life skills (managing appointments, completing household chores, following 
medication schedules, and adhering to homework schedules). The researchers indicated that twenty-two students 
(18 male and 4 female) demonstrated occupational performance gains eight weeks after training as measured by the 
Canadian Occupational Performance Measure (COPM) - a semi-structured interview assessment (test-retest 
reliability .89 and internal consistency .71) used to determine a client’s areas of disability. Performance on and 
satisfaction with five self-determined areas of need in everyday life tasks were rated and scored. The COPM was 
administered prior to PDA training and again eight weeks after PDA training. Each participant’s pre-and-post 
intervention ratings on five self-identified tasks were analyzed using paired samples t-tests. Results revealed a 
statistically significant improvement in performance and satisfaction with performance following PDA training. 
Additionally, eight weeks after PDA training, 82% of the students independently used the PDA to perform the 
following tasks: set reminder alarms, repeat event reminders, and make calendar and address book entries. 

Mosoco 


In an attempt to provide social skills support and help children with ASDs learn to apply the Social 
Compass (a social skills curriculum), researchers created a mobile social compass (MOSOCO). The augmented 
reality tool was designed to provide mobile and interactive social skills instruction that assisted children ages 8-11 
with aspects of the Social Compass Curriculum: appropriate eye contact, space and proximity, initiation of an 
interaction, asking questions, sharing interests, and terminating a conversation (Escobedo, Nguyen, Boyd, Hirano, 
Rangel, Garcia-Rosas, Tentori, & Hayes, 2012). A mixed-methods approach was used to analyze notes, interviews, 
and video transcripts collected over a seven week period of use during lunch time and recess. The researchers used 
open coding and a multi-phase affinity analysis to identify themes that surfaced as student used MOSOCO. Results 
were promising: students with an ASD spent more time engaged with others while they used MOSOCO, and they 
embraced MOSOCO as a supportive tool for social skills development. They learned to use the system quickly, their 
confidence increased, and the system provided the repetition students needed to leam to apply the Social Compass 
Curriculum. In addition, social mistakes (lack of eye contact, stems like rocking, jumping, noises, and abrupt 
conversation endings) were reduced by 56%. More research is needed to determine how well social skills assistance 
through MOSOCO generalizes to other contexts with larger numbers of students (only twelve students participated 
in this study, three with an ASD, and ten neurotypical peers who were invited to be potential interaction partners). 

Embedded Social Network to Support Daily Activities 

Mind blindness (the inability to empathize with others and understand their perspective, needs, thoughts, 
and feelings) is common in children and adults with ASDs, and it causes misunderstandings in social settings 
(Baron-Cohen, 1995; Baron-Cohen, 2008). Many children and adults with an ASD do not understand their own 
feelings and thoughts and have difficulty ascribing mental states to themselves and other individuals. They may not 
make correct inferences because of their view of the world, and they may misinterpret a situation or an action as a 
result. These misunderstandings can cause a great deal of frustration and isolation and make social acceptance 
difficult or impossible. As children in the spectrum reach maturity, many are unable to transition into adulthood 
with the skills necessary for independent living and successful employment; as such, their parents continue to 
provide daily assistance with schedules, security, and reminders to help them complete everyday tasks (Hong, Kim, 
Abowd, Arriaga, 2012). These issues prompted researchers to propose the concept of a SocialMirror, a life skills 
support tool connected to an online social network and embedded into an everyday object in the home (mirror). 

This tool would allow the user to take pictures and send those along with requests for assistance and calendar 
information to an online network of trusted advisors who would supply advice on self-care, professional etiquette, 
and social rules. Research on the concept was conducted in three phases which included participant observation, 
semi -structured interviews, and homogenous focus group interviews. A video prototype depicting three usage 
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scenarios was implemented and used in focus group discussions. The benefits: collaboration among caregivers, 
increased independence for young adults with ASDs, and in-situ feedback from a distributed network of human 
experts who function as coaches, guides, and facilitators. Concerns regarding the SocialMirror: conflict resolution 
among human experts supplying advice, information safety, and privacy. The concept of a SocialMirror was 
explored in a formative design exercise conducted with 12 young adults with an ASD and 16 caregivers. 

Interactive Visual Supports 

Visual support is provided to children with ASDs in order to reduce latency between transitions, promote 
independence, reduce verbal and physical prompts, reduce anxiety, provide predictability and structure, manage 
activities, complete tasks, clarify directions, and support communication (Cramer, Hirano, Tentori, Yeganyan, & 
Hayes, 2011; Dettmer, Simpson, Myles & Ganz, 2000). Children with an ASD have varying degrees of impairment, 
and as a result, their visual support needs to be customizable, flexible, easy-to-create and update, and easy to 
assemble and distribute. Several options are available for visual support and communication. They include the use 
of a visual communication system like PECS (Picture Exchange System). Teachers and caregivers use Boardmaker 
or other software to select and customize images from a preprogrammed image library. The images are printed, 
laminated and placed in binders or on communication boards to help students make requests, initiate conversation, 
recognize objects, or plan activities. Sophisticated augmentative communication devices convert images and image 
sequences into speech. These systems (i.e., Dynavox System, GoTalk, Activity Pad) are available for users who can 
afford the equipment and the training required. Hayes, Hirano, Marcu, Monibi, Nguyen & Yeganyan (2010) 
describe results from a qualitative study examining three interactive visual support prototypes: Mobile 
Communication Tools (Motocos), Interactive and Intelligent Visual Schedules (vSked), and Automatic Recording of 
Everyday Images (SenseCam). Through participatory design sessions, focus group discussions, interviews, and 
fieldwork, the authors determined the design guidelines needed for effective interactive visual supports. Their 
findings indicate that the following are needed: flexibility, customizability, and adaptability. The systems should 
support IEP goals, track the student's progress, capture data that assists with the diagnosis of behavior, and support 
communication between caregivers and children. 

Multitouch Tablets 

In the last decade, the number of computer-based interventions designed for children in the spectrum has 
increased (Hourcade, Bullock-Rest, & Hansen, 2012). More and more caregivers and teachers are seeking to change 
the outcome for children in the spectrum by gravitating toward flexible, portable, and easy-to-use tablets and apps 
that assist with time management, organization, communication, academic, and leisure skill development. Tablet 
devices are small and portable; they generate a great deal of student interest, support learning and communication, 
and facilitate the development of social skills (Cihak, Fahrenkrog, Ayres, & Smith 2010; Kagohara, 2010; 

Retherford & Sterling-Orth 2009). Hourcade, Bullock-Rest, & Hansen (2012) found that applications they 
developed for a multitouch tablet had a positive effect on students’ social behavior and expression, promoted 
creative expression, collaboration, compromise, and emotional understanding. Applications were developed through 
work with twenty-six children in the spectrum, their teachers, and other stakeholders. After spending thirteen two 
hour sessions at an after school program and an elementary school, researchers iteratively developed multitouch 
applications for their tablet computer system that could create drawings, author music, create visual puzzles, and 
model emotions. 


Technology Use and Children’s Software Preferences 

In order to move children with ASDs toward independent and productive adult lives, it is necessary to 
explore evidence-based interventions and those that show promise. Technology tools are promising, because 
children with an ASD show an interest in technology; they learn to use computerized tools easily, enjoy controlling 
the devices, excel at visual presentations, and find technology tools engaging (Escobedo, Nguyen, Boyd, Hirano, 
Rangel, Garcia-Rosas, Tentori, & Hayes, 2012; Hourcade, Bullock-Rest, & Hansen. 2012; Tartaro & Cassell, 2007). 
In addition, a number of researchers using technology tools with children diagnosed with ASDs have reported that 
students are motivated by technology and improve social skills, communication, turn-taking, conversation, and play 
skills (D'Ateno, Mangiapanello & Taylor, 2003; MacDonald, Clark, Garrigan & Vangala, 2005; Tartaro & Cassell, 
2007). 
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Hourcade, Bullock-Rest, & Hansen (2012), indicate that “Technologies can help us learn more about 
children with ASD, how their minds work, and how they relate to the World.” With this knowledge, user-centered 
tools can be developed which motivate, educate, and support children with an ASD. Observing students with an 
ASD to ascertain their preferences, needs, and issues provides developers with a feasibility analysis which can be 
used to explore the variability in the Autism Spectrum and build tools that are capable of addressing problem areas. 
With this in mind, an exploratory study was undertaken to discover interest in, and reactions to computerized tools 
and Computer Assisted Instruction. 


Theoretical Foundation 

Constructivism is the theoretical foundation for this investigation. In Constructivism, individuals create 
their own unique meaning of the world. The experiences and knowledge of the individual along with his first-hand 
interaction with objects and concepts in the environment allow him to expand his perception, change his opinion, 
and integrate new skills and concepts. Constructivist learning reverses the role of the student and the teacher. Instead 
of a teacher-centered environment where knowledge is dispensed, a student-centered setting emerges where 
knowledge is developed through collaboration, practice, and participation. The instructor becomes a guide, 
facilitator, or coach, and the student becomes an active player who explores, discovers, and builds his unique 
knowledge. Interaction with technology allows the user to create individualized knowledge constructs that form his 
reality. The learner is active and participatory with technology, and he manipulates functions within the software to 
solve problems, advance to higher levels, examine new concepts, and gain new insight. 

Purpose 

The purpose of this exploratory inquiry was to discover children’s interest in and reaction to computerized 
tools and Computer Assisted Instruction. Interest in software was determined by the amount of time children spent 
using software and hardware, interviews with children, and requests for specific software titles. The questions: 

1 . ) What software titles are requested most frequently by children in the laboratory? 

2. ) Why are software titles requested? 


Method 


Participants 

Five children (3 females and 2 males) participated in this study; they were between the ages of 8 and 12. 
Each child had a primary diagnosis of ASD; two children were diagnosed with Autism, one was diagnosed with 
Asperger Syndrome, and two were diagnosed with PDD-NOS. Diagnoses were made by a Pediatric Neurologist, an 
MD with Board Certification in Neurodevelopmental Disabilities, the Child Study Center (Fort Worth), and the UNT 
Health Sciences Center. Parents completed informed consent and background forms and participated in interviews. 
Parents indicated that the first problems they noticed in their children were language delays, early learning that 
stopped, and sensitivity to sound and touch. Parents described the therapies they used, whether or not a family 
history of ASDs was present, their concerns, their educational background, and the academic challenges of their 
children. The children attended a technology lab to address the academic and social skills challenges reported by 
their parents. 

Procedure 

Each child attended a special technology laboratory for 2-3 hours each week during the spring of 2010. 
During their weekly visit to the laboratory, each child had the opportunity to use different software packages and 
tools; their reactions to the software and their verbal feedback were used to determine their preferences. At each 
visit, each child was presented with five technology activities, and they had the opportunity to select the order of 
these activities. As each child used software, his/her reactions to the software, their performance in the program, and 
the amount of time spent in each program were recorded. Interviews conducted with each child's parent were used to 
determine home technology use and collect parental observations about the child and his/her use of technology. 
Themes emerged from the parental interviews. Interviews with children were recorded and used to determine their 
level of interest in different software and the features they preferred. Data was analyzed using SPSS and Excel. 
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Findings 

Background and Demographics 

Parents of the lab attendees indicated that their child was attending Speech and Occupational Therapy 
sessions - 100% of the parents were engaged in these therapies. ABA, Social Skills Instruction, and Music Therapies 
were the next most frequently used therapies (40%). Parents indicated that there was a family history of ASDs 
(60%), and the most common parental concerns were: motor skill development (60%) followed by attention deficit 
and poor muscle tone (60%), anxiety and the need for self-help skills (40%). Academically, reading comprehension 
was a problem reported by (60%) of the parents and 80% of the parents reported problems with mathematics: 
division, fractions, multiplication, addition and subtraction. While in the lab, each child used CAI Software designed 
to build skills in mathematics, reading, and social skills. The parents reported that they had either attended college, 
had advanced degrees, or were working on a terminal degree. They reported that their spouses had attended college 
or completed college. 

Software Preferences 

Software preferences surfaced: Nickel Takes on Disrespect (Thinking Publications), Nickel Takes on 
Stealing (Thinking Publications), Mini Mystery Readers (Remedia Publications), Read, Write, & Type! (Talking 
Fingers, Inc.,), Words & Music (Topologika Software - Marion Broadie), and others. The children requested these 
pieces of software most often, and they worked on these much longer than other software. The packages had the 
following features: realistic content, combinations of audio and video, animation that could be invoked by the user, 
exaggerated characters, vivid color and graphic elements, clear examples, easy to follow rules and program 
instructions, an element of adventure, immediate feedback, the ability to take different paths through the software, 
personalization, and record keeping options. Some comments participants made to describe the software, their needs, 
and their experiences are listed below: 

Show Me Math: “[It] shows the right number at the right time. This could be a perfect one for kids who 

struggle.” 

Story Town: “I want to see a few more stories before I create my own.” 

Mini Mystery Readers: “I remember the most important parts on each page.” 

Nickel Takes on Disrespect: “Kinda lame, isn't it?” reference to use of sound in a segment of the program 

Nickel Takes on Stealing: “If you do something right, Paco will go up!” reference to making progress in the 

program 

Most of the time lab participants responded with “I’m not tired this is fun.” and “Oh no, 1 don't want to go.” 
However, there were times when the children had overly-active sensory systems (hypersensitive) and responded to 
sounds by covering their ears. When this occurred, the volume was adjusted to accommodate them and provide more 
comfort. Headphones were used if the child requested those, and the volume was turned off and gradually increased 
to the point where the child was comfortable. Additionally, children received frequent breaks during which they 
drew pictures, painted or played board games. Exercise balls were available for children to receive vestibular 
stimulation when they desired it. All of these were designed to reduce stress and anxiety and reduce 
hypersensitivity. 

Each child used a variety of software and hardware at home: Nintendo DS, e-Books, Wii, Laptop 
Computers with Internet Access, and home-schooling software curricula. Parents reported BrainPop (FWD Media), 
steaming video from NOVA (PBS Series) and the Discovery Channel, Magic School Bus Games (Scholastic), and 
Digi Games (Dress Up 121.com) as the most frequently used online resources at home. Parents indicated that the 
following software was also used at home: Math Whizz (Whizz Education), JumpStart World and JumpStart Games 
(Knowledge Adventure), Where in the World is Carmen Sandiego? (The Learning Company), Mavis Beacon 
Keyboarding Kidz (Borderbund), Click n Spell (ClickNKIDS), Aladdin Chess Adventures (Disney), and others. The 
parents indicated that their children did not like boring games, little control over the software, inconsistent 
presentations, confusing rules, and games whose difficulty increased too rapidly. In the lab, the children became 
impatient and sad when software took too long to load, had a lengthy orientation, or had unpredictable results. 

Software used in instructional environments that address the needs and issues of children in the Autism 
Spectrum can increase student learning, offer alternative ways of delivering instructional content, provide a more 
enjoyable educational experience for the learner, mitigate skill deficits, provide the opportunity for students in the 
spectrum to become full partners in their educational experience, facilitate transfer (Bosseler & Massaro, 2003; 
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Hetzroni & Tannous, 2004; Tartaro & Cassell, 2007), and serve as a form of positive reinforcement. Technology is a 
powerful force that can be motivating for students in the spectrum because of its visual presentation. Many students 
in the spectrum are visual learners. In this study, all parents reported that their children absorb material quickly and 
more completely when videos were used to explain content. The combination of visual and auditory (multi-modal) 
presentations should not be overlooked for content delivery. Technology can provide a sense of accomplishment, 
teach rules, provide access, increase productivity, and facilitate connections between learners which bridge time and 
space. 


Characteristics of Software Preferred by Students 


Intuitive Learning Opportunities 

Short Games without Repetition 

A Gradual Increase in the Difficulty of the Program 

An Adventure Element 

Entertaining Video Segments 

Auditory and Visual Instructions 

Animation Initiated by the Student 

Easy to Follow Rules and Instructions 

Clear Examples 

Variety 



Conclusion 

This inquiry provided basic information for the selection and evaluation of educational software by parents, 
teachers, and other professionals working with children in the Autism Spectrum. Information from this study can be 
used to create experimental designs which can be used to determine the effectiveness of software which contains 
user preferences. In this study, a variety of preferred software packages were identified for children 8-12 years of 
age with ASDs. Further exploration of these packages is needed, as additional research is needed in the area of 
assistive, adaptive, and instructional technology and its implementation for children in the Autism Spectrum. 
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Introduction 

The practice of science revolves around argumentation, defined as the ability to support claims about 
natural phenomena with evidence and the ability to critically evaluate and interpret the claims of other scientists 
(Abi-El-Mona & Abd-El-Khalick, 2011; Osborne, 2010). Argumentation is crucial not only to communication in 
science, but also to understanding scientific phenomena and solving scientific problems (Ford, 2012; Jonassen & 
Kim, 2010; Kuhn, 2010). Through argumentation, scientists can refine their solutions to address problems more 
thoroughly and appropriately, and incorporate useful ideas from other scientists (Osborne, 2010). 

Central to the ability to argue about scientific problems is argument evaluation ability, defined as the ability 
to judge the degree to which a claim is reasonable based on a weighing of the associated claims and premises (Ford, 
2012; Perelman & Olbrechts-Tyteca, 1958; van Eemeren, Grootendorst, & Snoeck Henkemans, 2002). Criteria by 
which individuals evaluate arguments can vary (Abi-El-Mona & Abd-El-Khalick, 2011; Driver, Newton, & 

Osborne, 2000), and the resulting evaluations can thus vary in quality (Glassner, Weinstock, & Neuman, 2005). 

Scientific Argumentation Ability as a Core Ability of 21st Century Citizens 

Scientific argumentation ability is important for all students to develop, because in the 21 st century, all 
citizens in democratic societies will need to make informed decisions about such public policy issues as local 
pollution and the safety of genetically modified foods (Kolstp, 2001; ten Dam & Volman, 2004). Citizens will only 
be able to do so if they can maintain a healthy skepticism and critically evaluate claims about scientific issues made 
by scientists and others (Ford, 2012; Halonen, Dunn, Baker, & McCarthy, 2011). 

Scientific Argumentation Ability and Middle School Students 

It is well documented that middle school students face challenges in argumentation (Belland, Glazewski, & 
Richardson, 2008; Kuhn & Udell, 2007; Sandoval & Reiser, 2004; Yoon, 2011). The skills involved in 
argumentative discourse are difficult to acquire (Kuhn & Udell, 2007). Gaining higher-order thinking abilities can 
be especially difficult for lower-achieving students, and as a result many instructors do not bother attempting to raise 
lower-achieving students' higher-order thinking abilities (Zohar & Dori, 2003). But lower-achieving students are 
arguably the students who need help developing higher-order thinking abilities the most. 

A Possible Solution: Instruction Centered on Socio-Scientific Issues 

If one wishes to help middle school students develop scientific argumentation ability, it is crucial to give 
them something about which to argue (Jonassen & Kim, 2010). Socioscientific issues (SSIs) are complex, 
multifaceted scientific problems that involve social, ethical, political, and moral issues (Walker & Zeidler, 2007; 
Zeidler, Sadler, Simmons, & Howes, 2005). SSIs are ill-structured, meaning that they have multiple potentially 
appropriate solutions, and there are many ways to arrive at solutions (Jonassen, 2010). Since SSIs are usually ill- 
structured, debatable, and argumentative in nature, they require that students perceive and evaluate values from 
multiple perspectives in the process of decision-making (Lee, Chang, Choi, Kim, & Zeidler, 2012). In short, the only 
way to judge the appropriateness of a problem solution is through the tools of argumentation (Jonassen, 2010). 

To promote student engagement, it is important to ensure that SSIs are personally relevant (Barab, Squire, 

& Dueber, 2000; Fredricks, Blumenfeld, & Paris, 2004; Tal & Kedmi, 2006). This can be done in part by selecting 
SSIs that are locally relevant and that student perceive relate to their current and future lives (Barab et al., 2000; 
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Nason & Woodruff, 2003). One way to use SSIs in middle school instruction is to make an SSI the central problem 
in a problem-based learning unit (Hmelo-Silver, 2004). In PBL, to solve the central problem, students need to define 
the problem, find and use the related information, and develop a possible solution to the problem (Barrows, 1985; 
Hmelo-Silver, 2004). 

Scaffolding 

Because middle school students struggle in the process of argumentation, it is crucial to support them in 
this process as they engage with SSIs (Belland et al., 2008; Jonassen & Kim, 2010; Kuhn, 2010). Scaffolding is 
support that allows students to engage in and gain skill at tasks that are beyond their unassisted capabilities (Wood, 
Bruner, & Ross, 1976). While scaffolding originally connoted support by a more capable adult, computer tools can 
also scaffold students by providing needed structure for difficult tasks (Belland, In press; Kyza & Edelson, 2005; 
Quintana et al., 2004; Reiser, 2004). Computer-based scaffolding has been found to improve students' ability to 
create evidence-based arguments (Belland, 2010; Belland, Glazewski, & Richardson, 2011; Cho & Jonassen, 2002). 

In our previous research, we learned that generic computer-based argumentation scaffolds can have 
differential impacts on middle school students of varying achievement levels. Specifically, lower- and average- 
achieving students benefitted more than higher-achieving students (Belland, Glazewski, & Richardson, 2011; 
Belland, 2010). We then redesigned our scaffolds to improve efficiency and efficacy. In this paper, we investigate 
the scaffolds’ use by students investigating a different SSI problem. 

Research Questions 

How do computer-based argumentation scaffolds influence argument evaluation ability? 

a. What is the overall influence? 

b. Does the influence vary among student subgroups? 

Method 


Setting and Participants 

This study was conducted in a Title 1 middle school (44% of students received free and reduced lunch) 
located in a small, rural community in the intermountain west. Sixty-nine 7 th grade students in three class sections 
participated. The teacher had 22 years of experience in middle school science education but no experience in 
inquiry-based instruction. He taught all students from three class sections through the unit. 

In Fall 2011, the teacher led three class sections of 7th grade science through a 3-week unit. Students 
worked as groups in this unit, each group had 3-4 students and was assigned with different stakeholder positions 
(e.g., environmentalists, common citizens). Students needed to (a) go to various locations along their local river to 
conduct several water tests; (b) analyze the water quality data (c) create a solution to optimize the water quality of 
the river, and (d) present an evidence-based argument supporting their solutions. Students in experimental condition 
had access to the Connection Log, while students in the control condition did not. Based on their stakeholder 
positions, students might focus on different aspects of water quality. For example, students who represented 
fisherman might be most concerned with the amount of dissolved oxygen in the water, so they might provide a 
solution to increase the amount of dissolved oxygen in the river if they find that the dissolved oxygen level is low. 
To solve the problem, students might create different solutions that can only be evaluated by the evidential support 
they use to back up their solutions. 

Independent Variable 1 - Scaffold Condition 

There were two levels of scaffold condition: exposure to computer-based argumentation scaffolds 
( Connection Log) or not. Two class sections were randomly assigned to use the Connection Log during the unit. The 
other class section completed the same unit without the support of the Connection Log. 

Students in the two experimental sections used computer-based scaffolds called the Connection Log. The 
Connection Log is a database-driven website which is designed based on a conceptual framework to scaffold middle 
school students’ creation of evidence -based arguments (Belland, Glazewski, & Richardson, 2008). In our previous 
empirical studies, by using the Connection Log, average-achieving experimental students performed significantly 
better in argument evaluation than average-achieving control students (Belland, 2010). In addition, lower-achieving 
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experimental students performed significantly better in argument evaluation than lower-achieving control students 
(Belland, et al, 2011). 

In 2010-201 1 we revised the Connection Log based on expert reviews and one-to-one evaluations with 
target students. The scaffolds consist of five stages, each representing a step in a generic process of construction of 
evidence -based arguments (Belland et al. 2008): 

1 . Define the Problem : Students define the problem they need to solve by stating three major components of a 
problem definition: stakeholder, what is happening, and how it affects stakeholder. 

2. Determine Needed Information: In this stage, students decide what information they need to find and 
where they can find the information. Each student will be assigned to find different information. 

3. Find and Organize Needed Information: Students find information and save it in the Connection Log as 
their evidence, then organize information under different categories. 

4. Develop Claim: Based on the information they find, each student create a possible solution of the 
problem, then discuss it as group and create a group solution. 

5. Link Evidence to Claim: Students choose relevant information they saved in the Connection Log as their 
evidence, and use them to support their group solution. 

Each stage consists of 3-4 steps: students first perform a task individually, and then discuss with 
groupmates. They need to come to consensus about what they articulate to go to the next step. Students are free to 
return to the pervious step to revise their information. 

Control students did everything that experimental students did. except that they did not have access to the 
Connection Log. They used computers to analyze water tests data, search information online and saved their notes 
by using word processing software. 

Independent Variable 2 - Prior Science Achievement 

Students’ test scores were matched with their science grades to date. Students who received A’s were 
deemed higher-achieving. Students who received B’s or lower were deemed lower-achieving. 

Data Collection 

To examine students’ argumentation ability change during the unit, all students from three class sections 
took argument evaluation pre and post-tests which were originally created by Glassner, Weinstock, and Neuman 
(2005). During the tests, students read argumentation scenarios first and rate assertions which can explain or provide 
support to the goal of the argument in the scenarios. We adapted the test with permission for our previous studies 
(Belland. 2010; Belland et al., 201 1). The coefficient alpha's of the revised test ranged from .70 to .82 in those 
studies. In the revised argument evaluation test, there are four units of questions. In each unit, students read 
scenarios in which a claim (e.g., the city should build a skateboard park) is made, and two supporting statements 
(e.g.. Skateboarding is a fun activity for kids to do after school; 50% of the kids in this town skateboard, and 
skateboarding after school at a skateboard park would keep them out of trouble) are provided. Students then rate 
how well each statement proved the claim from three choices: “doesn't help”, “helps a little”, and “helps a lot”, and 
choose which statement best proved the claim. There are 24 questions in both the pretest and the posttest. 

Procedures 

The pretest was given on the day before the unit started. Students from all three sections engaged in the unit 
for 50 minutes per day. There were 6-7 groups in each class section, and each group had 3-4 students. All small 
groups were constructed to consist of (a) students representing the ability range for the class period, and (b) when 
possible, an equal number of male and female students. 

On Day 1, a guest speaker spoke about the history of the local river. On Day 2, the teacher introduced (a) 
the purpose of the unit; (b) content knowledge of water quality; (c) the problem students needed to solve during the 
unit; (d) what students needed to present at the end of the unit and each group's stakeholder position. All students 
had access to a portable computer during the unit. On Day 3, students began to work in small groups and discussed 
characteristics of their stakeholder. On Day 4, students went to the river to collect water quality data at three 
different spots. Then in Days 5-14, students worked in small groups. They analyzed data from the water quality 
tests, searched for related information online, discussed and created their solutions, and used evidence to back up 
their solutions. At the end of the unit, students presented their solutions to the county commissioner. The day after, 
students took the posttest. 
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Analysis Strategies 

Since we assigned intact classrooms to treatment conditions, we first used nested ANOVA to analyze the 
data. There was no significant nested effect of scaffolds condition on argument evaluation ability, F (1,56) = 0.14, 
p=0.71. Therefore, a two way ANOVA was used to compare pretest to posttest gain scores with scaffold condition 
and prior science achievement as independent variables. 

Results 


Manipulation Check 

There was no statistically significant difference in students’ science grades between the three class sections, 
F (2, 64) = 0.64, p = 0.53. There was no statistically significant difference in students’ argument evaluation pretest 
score between the three class sections, F (2, 60) = 0.23, p = 0.80. 

What is the overall influence of the Connection Log on argument evaluation ability? 

The reliability coefficient (Cronbach’s alpha) for pretest and posttest was 0.84 and 0.88, respectively. It 
shows that the argument evaluation tests have good internal consistency (Kline, 1999). Two variables (scaffold 
condition and prior achievement) were included in the ANOVA model. The outcome variable was gain from pretest 
to posttest. The main effect of the Connection Log on argument evaluation ability was not significant, £(1, 53) = 
3.6, p = 0.063. 

Does the influence of the Connection Log vary among student subgroups? 

There was a significant disordinal interaction between scaffold condition and prior achievement, F( 1, 

53)= 14.53, p<.001. See Figure 1. A test of simple main effects indicated that lower-achieving experimental students 
gained significantly more than lower-achieving control students, £><0.05, £5=1.26. Lower-achieving experimental 
students gained 0.775 standard deviations from pre to posttest, while lower-achieving control students lost 0.28 
standard deviations from pre to posttest. See Figure 2. 

Estimated Marginal Means of GainScore 



Fig. 1 Interaction effect of scaffolds condition and prior science achievement 
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Discussion 


Of note is that the Connection Log was used in this study in conjunction with students’ investigation of an 
entirely different problem - related to watershed science rather than genetics - from that used in Belland (2010) and 
Belland et al. (201 1). Our findings echo previous research that indicated that the Connection Log led to higher 
argument evaluation scores among lower- and average-achieving students, but not among higher-achieving students 
(Belland, 2010, Belland. et al., 201 1). But the effect size is considerably larger - twice as large. Lower-achieving 
students in this class were so designated because of their prior science grades. So they clearly had the most room to 
improve. 

The results are interesting in that they provide evidence that an intervention designed to raise a particular 
higher-order thinking skill among middle school students worked better among lower-achieving students than 
among higher-achieving students. These results indicate that it may be appropriate to focus more interventions on 
helping lower-achieving students to improve their higher-order thinking abilities (Zohar & Dori, 2003). This is 
important because all citizens in the 21st century need to be able to argue about scientific issues (Kolstp, 2001; ten 
Dam & Volman, 2004). 



QPreTest 

SPostTest 


Fig. 2 Changes among lower-achieving students in control and experimental group on pre and post tests 


The results also indicate that the redesign of the Connection Log likely improved, or at least did not lead to 
a degradation of, its capacity to increase lower-achieving students' argument evaluation ability. Further research 
along these lines is needed both with similar units and different units. 

It is interesting that the Connection Log worked in conjunction with a unit on an entirely different 
topic/subject from that of our previous studies (Belland, 2010; Belland et al., 201 1). This is important because to 
promote engagement it is important to ensure that PBL problems are potentially authentic (Barab et al., 2000; Nason 
& Woodruff, 2003). The unit in the previous studies focused on genetics (the human genome project), while the unit 
in this study focused on watershed science (water quality in a local river). 

Limitations and Suggestions for Future Research 

This paper relies on one data source, which limits its descriptive power. Other data collected in the project, 
including video observations, interviews, and database information, are still in analysis, and when analyzed should 
shed further light on the performance of the students. 

The sample size, along with the odd number of class sections, reduced statistical power. Few principals will 
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allow individual K-12 students to be randomly assigned to treatments, and thus we needed to randomly assign class 
sections to treatments. This does negatively impact the internal validity of the study, but by keeping students in their 
assigned class sections, we studies them in their natural setting, which boosted ecological validity. Further research 
is needed with a larger sample size, and ideally, an even number of class sections. Also, further research with the 
Connection Log or similar scaffolds in conjunction with different units would contribute to a further understanding 
of generic scaffolding. 
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Abstract 

This mixed methods research examined the relationship between the students’ initial posts (quality, length, 
and posting day) and other students’ responses (quality, length, and number of responses). The findings indicated 
that the quality of initial posts was positively related to the quality and the number of responses. In addition, the 
posting day of initial posts was negatively related to the quality and the number of replies. However, the length of 
initial posts had no relationship to responses. 


Introduction 

Online discussion is a powerful teaching and learning tool to facilitate students' knowledge construction 
and active interactions in online courses. Various instructional interventions have been employed in an effort to 
improve online discussion quality and interactions, such as types of discussion topics (e.g., Kanuka, Rourke, & 
Laflamme, 2006; Wang, 2006; Han, Cheon, & Chung, 2010), assigned roles of discussion leaders (An, Shin, & Lim, 
2009; Hew & Cheung, 2008; Wever, Keer, Schellens, & Valcke, 2009), structured discussion groups (e.g.. Brooks & 
Jeong, 2006; Jeong & Joung), and structured discussion questions (e.g., Han & Cheon, 2011). Furthermore, 
previous studies have proposed that students’ individual differences, such as GPA, major area of study, or other 
characteristics (e.g., Krentler & Willis-Flurry, 2005; Nussbaum & Bendixen, 2003; Spatariu, Quinn, & Hartley, 
2007), would also affect students’ performance in online discussions. 

However, little research has indicated how students’ initial posts may influence the responses and 
interactions of other students. The purpose of this study was to investigate the relationship between the students’ 
initial posts (quality, length, and posting day) and received responses (quality, length, and number of responses). 

The results suggested some additional factors that may also impact aspects of online discussion qualities and 
interactions. 


Research Methodology 


Participants 

Thirty undergraduate students voluntarily attended an online discussion activity in an online course entitled, 
“Computing and Information Technology”, at Texas Tech University in Fall, 2011. For their participation, each 
participant received five extra points. 

Procedure 

Based on previous research (e.g., Tenenbaum, Naidu, Jegede, & Austin, 2001; Kanuka et ah, 2006), this 
study designed a debating discussion topic about societal attitudes toward new technology and students’ own 
opinions concerning the roles of technology in Education. To keep the size of the discussion group small (e.g., Du, 
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Zhang, Olinzock, & Adams, 2008), students were randomly put into four groups to discuss the same topic in the 
span of eight days. They were required to reply to at least two other initial posts. 

Instruments 

Each post’s quality, length, and posting day were measured and recorded. The quality of posts was coded 
by Interaction Analysis Model (IAM) (e.g., Wever et al., 2009; McLoughlin & Mynard. 2009). IAM describes five 
phases of negotiating meaning and knowledge co-construction in a collaborative online discussion (Gunawardena, 
Lowe, & Anderson, 1997). The five phases are: (a) sharing/comparing of information (PHI), (b) 
discovery /exploration of dissonance/ inconsistency amongst participants (PHII), (c) negotiation of 
meaning/knowledge co-consttuction (PHIII), (d) testing/modification (PHIV), (e) phrasing of agreement and 
applications of newly constructed meaning (PHV ). Higher phase represents higher level of knowledge construction 
(LKC). Two coders coded all the posts independently and finally discussed and decided each post’s level together. 
Qualitative analyses were also used to discover keys of constructing high quality posts. The lengths of posts were 
measured by the word count. Since the discussion lasted eight days, the posting day were coded as one to eight from 
the first day to the last. 


Results 

A series of correlation analyses were conducted to analyze the relationship between the students' initial 
posts (quality, length, and posting time) and received responses (quality, length, and number of responses). The 
results (see Table 1) indicated that the quality of initial post was positively related to the quality of responses, r 
= .610 ,p< .001 and the number of responses, r = .327, p < .05. The posting day of initial posts was negatively 
related to the quality of responses, r = -.386, p < .05 and the number of responses, r = -.432, p < .01. However, the 
length of initial posts and responses had no relationship. The table 1 also showed additional results that the quality 
of initial posts was negatively related to the posting day and was positively related to the length. In addition, the 
quality, number, and length of responses were positively related with each other. 

Table 1. Correlation Matrix 



1 

2 

3 

4 

5 

6 

1 .Quality of initial post 

2. Posting day of initial post 

3. Length of initial post 

4. Quality of response 

5. Number of response 

6. Length of response 


-.434” 

.625** 

ns 

.610" 

-.386* 

ns 

.327* 

-.432” 

ns 

.405* 

ns 

ns 

ns 

.616** 

.456" 


*Correlation is significant at the 0.05 level. 

**Correlation is significant at the 0.01 level, 
ns. Correlation is not significant. 


Discussion and Conclusion 

The quantitative results of this study showed that the quality of initial posts played an important role in the 
quality and the number of responses. High quality initial posts led to higher quality and more number of responses. 
Moreover, the qualitative analyses found that the high quality posts could negotiate the meanings of discussion topic, 
identify areas of disagreement, summarize their metacognitive statements and show their new knowledge 
construction very well. They used popular topics, such as social networking, smart phone, iPad, or new movies, to 
explain their opinions and motivate peers’ discussion interests. In addition, including interesting questions in initial 
posts was another strategy to trigger other students’ thoughts and received more and higher quality responses. For 
example, one initial post which got many responses asked “. . .1 couldn’t believe people would actually fire you over 
the internet. But, is that the world is going to come too?...”. The quality of initial posts could be an important factor 
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to influence the interactions of online discussions. The findings suggested some good strategies for designing a high 
quality discussion topic as well. 

This study also indicated that later posts would receive less quality and total number of responses. Students 
may not have enough time to construct new knowledge and fully summarize their statements when they posted late. 
Thus, other students may miss their posts or choose others to response. Therefore, setting up another earlier 
deadline for posting initial posts, mentioning the importance of posting earlier in discussion policies, or sending 
students’ reminders might be the ways to help all the students post earlier. Interestingly, in this study, the length of 
initial posts was not a factor to affect other students’ responses. In other words, students would not choose to 
response only because the initial posts were long or short. 

In addition, for initial posts, the results of this study showed that the quality was positively associated with 
the posting day and the length of posts. Student who posted earlier might have more motivation and time to 
contribute high quality posts. The findings about responses were similar to the initial posts. 

The study findings indicated several factors that may also influence the quality and interactions of online discussions. 
Several implications and recommendations were suggested for online discussion design and practice. However, this 
study was conducted in a particular education setting with a small sample size and one type of discussion topics. 
Further studies could use a larger sample size and practice more types of discussion questions. 
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Abstract 

This study examined two types of discussion questions: (a) debating questions and (b) student-generated 
questions and the effect on students’ interactions during asynchronous online discussions in a Chinese online course. 
The quality, number, and length of students' replies were analyzed to study students’ interactions. The findings 
indicated that student-generated questions could improve both the quality and the number of students’ replies. 
However, the length of replies showed no difference between the debating questions and student-generated 
questions. 


Introduction 

Today, online discussion has become one of the most common interventions to facilitate students’ 
constructivist learning and active interaction in online courses. It is suggested that online learning environments are 
more student-centered compared with classrooms (Kassop, 2003). Especially, online discussion might be an 
effective tool in the Asian online classes where students are typically less verbal in public (Chang, 2001). 
Asynchronous discussion can increase their motivation to express freely (Teo, 2007). 

Discussion is a process that uses questions to drive thinking (Wang, 2005). A high-level question can 
stimulate mental activities and help students construct understanding (Chin & Langsford, 2004). Previous studies 
have proposed that the debating discussion questions had higher participation and cognitive presence ( Kanuka, 
Rourke, & Laflamme, 2006). However, the problems are that Chinese students may not know what to discuss; they 
may have difficulty formulating contributions or they may make uncritical comments (Guzdial & Turns, 2000). 
Moreover, students tend to only post the minimum number of messages required, particularly if participation is 
voluntary (Fung, 2004; Hara, Bonk, & Angeli, 2000). 

This study explored the use of two types of discussion questions, (a) debating questions and (b) student- 
generated questions and the effect on students’ interactions during asynchronous online discussions. The quality of 
students’ replies, the number of replies, and the length of responses were measured in this study to investigate which 
discussion question could improve students’ interactions. The results indicated considerable implications for 
designing effective online discussion activities. 

Research Methodology 


Participants 

There was 102 in-service teachers who enrolled in the online course entitled, “Digital Photography and 
Image Processing”, in Fall, 201 1 at a metropolitan university in Shanghai, China. The semester was five months 
long from Nov. 2011 to Feb. 2012. Students were all K-12 teachers. During the semester, they attended all the 
online discussion activities voluntarily. 
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Procedure 


This study designed two types of discussion questions: (a) debating question and (b) student-generated 
question. In order to keep the small discussion group size (e.g., Du, Zhang, Olinzock, & Adams, 2008), students 
were randomly assigned to four groups. The study consisted of two phases each lasting one month. In the first phase, 
students were asked to discuss a same debating topic about the roles of digital photography and image processing. 

In the second phase, all the students kept discussing in the same groups, but were asked to generate their own 
questions as their initial posts. Students were required to reply to at least two others’ initial posts in each phase. 

Instrument 

To analyze students’ interactions, the quality, the number, and the length of each reply was measured and 
recorded. The quality of students’ replies was coded by using Interaction Analysis Model (IAM) (e.g., Wever et al., 
2009; McLoughlin & Mynard, 2009). IAM is a content analysis model based on a constructivist paradigm designed 
to detect evidence of knowledge construction (Gunawardena, Lowe, & Anderson, 1997). It has five phases: (a) 
sharing/comparing of information (PHI), (b) discovery/exploration of dissonance/ inconsistency amongst 
participants (PHII), (c) negotiation of meaning/knowledge co-construction (PHIII), (d) testing/modification (PHIV), 
(e) phrasing of agreement and applications of newly constructed meaning (PHV ). Higher phase represents higher 
level of knowledge construction (LKC). One post may achieve several phases of IAM. The highest phase that one 
post achieved was used to identify its level. All the posts were coded by two coders and they decided each posting’s 
level together. The length of each reply was coded by word count. 

Results 

There were 32 in-service teachers who participated in both phases voluntarily. Totally, they contributed 58 
replies in the first phase and 79 replies in the second phase. Three Paired-Samples T tests were conducted to 
analyze whether the number, the quality, and the length of replies were significant different between debating 
question and student-generated question. The results revealed that student-generated discussion question ( M = 2.47, 
SD = 1 . 41 ) led to more number of replies than debating question ( M = 1.81, SD = 1.18), t = -2.10, p< 0.05. It ( M = 
2.00, SD = 1.20) also led to higher quality of replies than debating question (M = 2.73, SD = 1.43), t = -2.25, /;<(). 05. 
However, there was no significant difference between the lengths of replies (see Table 1). 

Tablel. Mean scores and results of Students’ Interactions 



Debating 

Discussion Question 

Student-Generated 
Discussion Question 



Mean (SD) 

Mean (SD ) 

p Value 

Number of Replies 

1.81 (1.18) 

2.47 (1.41) 

.044* 

Quality of Replies 

2.00(1.20) 

2.73 (1.43) 

.032* 

Length of Replies 

63.60 (33.76) 

55.57 (36.56) 

.366 


* p < .05 


Discussion and Conclusion 

Limited student participation in online discussions appears to be a persistent problem in this course. Less 
than half the students attended the discussions mainly because the discussion activities were not required. On the 
other hand, influenced by Confucian Heritage Culture, Chinese students are cultivated to revere authority, maintain 
harmony, and avoid conflicts (Chiu, 2009). It makes them appear less verbal in public (Chang, 2001). Therefore, in 
online courses, Chinese students may not know what issues to discuss; they may have difficulty formulating 
contributions or they may make uncritical comments (Guzdial & Turns, 2000). It is always a big challenge to 
motivate more Chinese students to join online discussions and help them improve critical thinking abilities. 
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With the limitation, this study found that student-generated discussion question led to a greater number of 
students’ replies than the debating discussion question. Normally, students tend to only post the minimum number 
of messages required, particularly if participation is voluntary (Fung, 2004; Hara, Bonk, & Angeli, 2000). In this 
study, each student was asked to reply to at least two other initial posts. However, for the student-generated 
questions, 32 students contributed 79 replying messages. On average, each student replied to more than two posts. 
The results indicated that Chinese students felt more comfortable to discuss peers’ questions. Consequently, the 
participation and interactions to discuss student-generated question were improved. 

The second finding showed that student-generated discussion question led to higher quality of students’ 
replies than debating question. In this study, each post was coded by IAM’s five phases from one to five. The 
average score of IAM is 2.5. For student-generated question, the mean score (M = 2.73) of the quality of students’ 
replies was above 2.5. Although previous studies have indicated that the debating discussion questions had higher 
participation and cognitive presence (e.g., Kanuka, Rourke, & Laflamme, 2006; Han, Cheon, & Chung, 2010), 
student-generated discussion question tended to be more effective for Chinese students to stimulate their thoughts. 
Thinking is driven by questions, not by answers (Elder & Paul, 1998). Student-generated questions can facilitate 
Chinese students’ efforts to summarize the course content, generalize the issues, post their own questions, and 
synthesize others’ opinions. It is an effective thinking process. Throughout this process, students are cognitively 
engaged in knowledge construction and the meaningful participation and interactions are promoted. 

However, the length of replies showed no difference between debating question and student-generated 
question. It was expected that students would contribute longer posts for student-generated question. However, the 
results indicated that there was no relationship between the length of posts and the quality of the posts. Students’ 
interactions were still improved by increasing the quality of responses and the number of responses through 
discussing student-generated question in this study. 

This study only measured two types of discussion questions. Further studies could examine additional 
questioning types to motivate more Chinese students attend online discussions, encourage their meaningful 
participation, and promote discussion interactions. Furthermore, this study could be conducted in other countries to 
see whether the culture differences or characteristic differences would influence the results. 
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Peer Assessment in Group Projects: The Team Member Evaluation Tool 

In professional environments, including business, military, and industry, a team based approach is often 
used in both product and process development. This team based approach is reflected in many graduate and 
undergraduate courses where projects, based on group work, are used as a means of assessing student learning 
outcomes. Unfortunately, a phenomenon referred to as social loafing, where individuals evade the expected team 
member workload, reduces the effectiveness of the group work strategy (Aggarwal & O'Brien, 2008). 

The Team Member Evaluation Tool (TMET) is a web based tool developed on an existing initiative used to 
reduce the presence of social loafing in group projects. Through a series of formulas and three inputs including (a) 
group project peer assessments, (b) group project instructor assessments, and (c) group member evaluations, 
individual team members are assigned a grade based on their individual input to the overall group project. This 
manuscript offers insight into this ongoing design and development study including future goals for the TMET. 

Purpose 


The problem of social loafing exists across almost every academic discipline including marketing and 
computer science among many others (Aggarwal & O'Brien, 2008; Pieterse & Thompson, 2010). An intuitive user- 
friendly web based tool that reduces the social loafing problem does not currently exist. The outcome of this project 
is two-fold in that it includes both the tool, the web based TMET, and research findings on its use in educational 
contexts. One portion of this ongoing study includes the design, development, implementation, and evaluation. 

Three instruments comprise the TMET input and the output reporting for students’ grades is flexible. The second 
portion of the project includes the ongoing research agenda and findings regarding the TMET use in online courses. 
Two surveys, pre- and post-group work, consisting of Likert-type items and open-ended questions, and student input 
data captured through the TMET, are used for ongoing research and refinement of the TMET. 

Two research questions were developed based on the purpose of the TMET and served to guide this 
ongoing study. First, what are the necessary components required for implementing a web based tool for the 
calculation of individual grades for group projects? This research question permitted the researchers to focus on the 
design and development of the tool, including testing and revision, based on user input. Second, how are individual 
grades for group projects perceived? This research question guided the implementation and evaluation. 

Related Literature 

The theory of planned behavior (Ajzen, 1991, 2005) may be used to describe team members' intent and 
behavior regarding the level of contribution on a group project. The theory of planned behavior is comprised of three 
constructs: attitude, subjective norm, and perceived behavioral control (Ajzen, 1991). These three constructs are 
influenced by individuals’ behavioral, normative, and control beliefs respectively. Ajzen (2005) suggests 
“behavioral intentions are assumed to follow from their beliefs about forming the behavior” (p. 126). Individuals 
develop beliefs regarding group work through group project experiences. These experiences may have been positive 
or negative leading to potentially biased and even incorrect beliefs. Whatever the underlying belief structure an 
individual has established, “once a set of beliefs is formed it provides the cognitive foundation from which attitudes, 
subjective norms, and perceptions of control - and ultimately, intentions and behaviors - are assumed to follow in a 
reasonable and consistent fashion” (p. 126). 

For many years group work has been an essential instructional strategy reflecting the modern corporate 
environment by encouraging discussions and debate which require idea justification, disagreement resolution, and 
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understanding of different perspectives (Webb, 1995). Group work and group projects, while providing benefits of 
socialization and negotiation, “can also be more comprehensive in scope than individual projects and thereby 
provide more realistic learning experiences for students” (Aggarwal & O'Brien, 2008, p. 256) However, the 
perception that social loafing exists in these group work contexts is pervasive among public university students and 
undermines the potential effectiveness of the strategy (Piezon & Ferree, 2008). 

Unsatisfactory group work experiences are often attributed to social loafing by group members (Mello, 
1993; Pieterse & Thompson, 2010; Piezon & Ferree, 2008; Strong & Anderson, 1990; Williams, Beard, & Rymer, 
1991; Webb, 1995). Researchers have questioned the merit of a group grade and suggest instead that the 
contribution of each student should be reflected in individual grades for group projects (Butcher, Stefani, & Tario, 
1995; Cheng & Warren, 2000). 

“One of the best approaches for measuring individual input on group projects is the use of peer 
evaluations” (Miglietti, 2002, p. 1 13). The contribution each team member exerts towards completion of the group 
project, captured through peer evaluations, can be used to calculate individual grades (Cheng & Warren, 2000; 
Dyrud. 2001). 

Methodology 

Design and development research offers researchers in instructional technology related disciplines the 
ability to study the processes required for creating a product or model (Richey & Klein, 2007). Type 1 research 
encompasses product and tool development while type 2 research addresses model development. The present study 
described here is considered a tool design and development research study, conducted in three phases, providing the 
researchers with the opportunity to design, develop, implement, and evaluate a tool intended to generate individual 
grades for group projects. 

The first phase of the TMET development resulted in a set of Microsoft Word document forms and Excel 
spreadsheets that were used to gather both class member and team member input along with instructor input to 
generate individual grades on group projects. In a face-to-face or blended environment students would either 
complete the group project peer assessment emailed by the instructor or complete the assessment made available 
through the learning management system. The instructor would then input this data into spreadsheets that would 
calculate each individual’s grade for a group project. 

The overall group project grade consists of two elements: (a) group project peer assessment (25% of the 
group project grade) and the group project instructor assessment (75% of the group project grade) as suggested by 
Conway, Kember, Sivan, and Wu (1993). The group project peer assessments are based on: (a) quality of the 
project, (b) quality of the project content, and (c) instructional value of the project. The group project instructor 
assessments are based on a rubric specific to the course requirements. These two grades, peer and instructor project 
assessment, serve to create the combined group project grade. 

Once the group project grade is derived, then next step is to calculate the individual group member grade. 
Each group member evaluates themselves and their group members based on eight five -point Likert-type rating 
items using a form referred to as the group member evaluation form. The rating items consisted of: (a) motivation, 
responsibility, and time management, (b) adaptability, (c) creativity and originality, (d) communication skills, (e) 
general team skills, (f) technical skills, (g) workload balance, and (h) leadership skills (Lejk & Wyvill, 2001, p. 63). 
An individual group member weighting factor is then used to determine an individual weighting factor using the 
following formula: individual weighting factor = individual effort rating / average effort rating (Conway et al., 1993, 
p. 50). The individual weighting factor is then multiplied to the group project grade resulting in the individual group 
member grade. The average effort weighting was the sum total of ratings divided by the number of members in the 
group. The total rating on the six rating items for an individual provided the individual effort weighting. Dividing 
the individual effort rating by the average effort rating provided an individual group member’ s individual weighting 
factor. The individual weighting factor was then multiplied to the group project grade resulting in the individual 
group member grade (Conway et ah, 1993). The second phase of this project eliminated the need to gather student 
input through forms created in Microsoft Word. Instead the process was “automated” through a series of web-based 
forms. Course rosters for each class were uploaded to a database. Class members were randomly assigned to groups 
and groups were assigned group numbers which assisted in populating the forms with student information. 
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Data Collection 


Details regarding the specific instruments and formulas for their use were described in the previous section. 
This section provides the TMET process as it was implemented with regard to data collection (see Figure 1). 
Students in two graduate courses and one undergraduate course began by completing the pre-group work survey 
within the first week of the semester. Group projects in the courses sampled were not altered in any way and groups 
were randomly created through a feature in the course management system. Upon completion of the group projects, 
each class member reviewed the other group projects using the group project peer assessment. This is a standardized 
project review instrument used across all courses. The instructor for each course also reviewed the group projects 
through the group project instructor assessment which was essentially aligned to the group project rubric for each 
course. Scores from the group project peer assessment (25% of the group project grade) and the group project 
instructor assessment (75% of the group project grade) were added to create the group project grade for each project. 
Students then completed the group member evaluation form. Through the formula described in the previous section, 
the individual group member grade was calculated. Finally, class members completion the post-group work survey. 




Individua 
I Group 
Member 
Grade 



Group Project 
Peer 

Assessment 

• 25% of Group 
Project Grade 




y 


Group Project 

Instructor 

Assessment 

• 75% of Group 
Project Grade 


o 


y 


Group Member Evaluation Form 

• Individual Weighting Factor = Individual Effort 
Rating / Average Effort Rating 

• Individual Weighting Factor is then multiplied to 
the Group Project Grade 


y 


Figure 1. The TMET process including the specific input forms. 

Data Analysis 

The pre- and post-group work survey instruments each consisted of 13 Fikert-type items based on a 5-point 
rating scale (1 = strongly disagree and 5 = strongly agree) and four open-ended questions. The 13 Fikert-type items 
were the same for both the pre- and post-group work instruments. The four open-ended questions on the pre-group 
work instrument were used to assess positive and negative aspects of group work and elicited descriptions of 
experiences of each. The four open-ended questions on the post-group work instrument were used to elicit detailed 
feedback regarding derivation of the group member grade and the individual grade for the group project. A total of 
52 students served as the study population and seven students chose not to participate. Of the remaining 45 students, 
five did not complete one or more instruments and were excluded from the study, providing a total of 40 students 
serving as the study sample. Items 3, 4, 6, 8, 9, and 10 were negatively worded and were recoded. To test for 
reliability, Cronbach’s alpha for the 13 Fikert-type items was calculated (a = .712) and was considered to be 
acceptable. A paired-samples t-test revealed a statistically significant positive increase in four items of the 13 Fikert- 
type items on the pre- and post-group work items (see Table 1). 
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Table 1. 


Significant results (n = 40) based on the Paired Samples T-Test. 


Survey Item 

Pre-Group 

Work 

Post-Group 

Work 

t 

P 

1 . 

I enjoy working in groups. 

3.41 (1.07) 

3.61 (0.95) 

-2.08 

.044 

3. 

I work best alone and prefer individual 
projects, (recoded) 

2.43 (1.05) 

3.63 (1.04) 

-2.64 

.012 

6. 

It seems like I always stuck in a group 
where 1 do all the work, (recoded) 

3.22 (0.95) 

3.52(1.00) 

-2.45 

.019 

11. 

Group members assist me on group 
projects in areas where I may be 
weak. 

3.48(1.07) 

3.70(1.13) 

-2.10 

.043 


While it is too soon to make any assumptions based on these current results, it should be noted that in 
earlier tests of the TMET, the majority of students disliked group work at the beginning of the semester; the most 
common reason was the nature of the group project grade. Based on the results of the current data analysis, students 
enjoy working in groups. Item 4, 1 feel I am graded unfairly in group projects, was found not to be significant. In 
earlier tests of the TMET a significant difference existed based on this item, possibly suggesting that repeated use 
(testing and refinement) of the TMET in related courses in the current institution/program offers students a specific 
process and procedure regarding how group project grades and individual grades on group projects are derived. 
Qualitative data suggest that the use of an individual peer grade for a group project may increase students’ 
preference for working in groups. Additional items of note, while not found to be significant, suggest the following. 

1 . Individuals that identified themselves as group leaders dislike group work more than those that did not 
consider themselves leaders. 

2. Group leaders completed more work for group projects than other members; indicated by both (sell and 
appointed ) group leaders and group members. 

3. Individuals that were rated low in contributing sufficiently to a group project in turn rated all group 
members as contributing equally. 

Conclusion 

The TMET will be presented along with the research findings to date. The Team Member Evaluation Tool 
(TMET) is a web based application used to reduce the presence of social loafing in group projects. Through a series 
of formulas and three inputs, individual team members are assigned a grade based on their individual input to the 
overall group project. The outcome of this project is two-fold in that it includes both the product, the web based 
TMET, and research findings on its use in educational contexts. 

One portion of this study includes the development, programming, testing, and revision of the TMET and 
instruments. The second portion includes the ongoing research agenda and findings regarding the TMET. Two 
surveys, pre- and post-group work, consisting of Likert-type items and open-ended questions, are used for ongoing 
research and refinement of the TMET. The problem of social loafing exists across almost every academic discipline 
including marketing and computer science among many others (Aggarwal & O'Brien, 2008; Pieterse & Thompson, 
2010 ). 
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Abstract 

The purpose of this study was to investigate teachers’ beliefs regarding the integration of technologies for 
learning following exposure to the 2011 K-12 edition of the Horizon Report. Teachers read the Horizon Report and 
then participated in an asynchronous, threaded discussion focusing on technologies they would like to see integrated 
into their classrooms and technologies they do not see possible in the near term. Qualitative methods were used to 
analyze the discussion. Findings include the teachers' desire to see mobile technologies integrated into their 
classrooms, and their belief that game-based learning will not be prevalent in their specific contexts. 


Objectives and Purposes 

Despite ever increasing access to technology in schools, there have not been corresponding increases in meaningful 
uses of technology in schools. Some researchers (e.g. Belland, 2009; Ertmer, 2006; Wachira, P., & Keengwe, J., 

201 1) propose that teacher beliefs are one of the reasons behind a lack of technology use in the schools. Teachers’ 
beliefs of the value or perceived usefulness of various technologies may be important elements to consider when 
adopting technologies for teaching and learning. The research reported in this paper was conducted with the 
purpose of understanding teachers’ beliefs regarding certain emerging technologies. 

The Horizon Report (New Media Consortium, 2011) is an annual report, which reports forecasted trends in 
technology use in Education. Different versions of the Horizon Report are produced each year to address varied 
educational contexts. The 201 1 edition of the Horizon Report for K-12 education was used in the present study, as it 
was the most current edition at the time the research was conducted. It includes predictions for technologies that 
will be adopted in schools in three categories: one year or less, two to three years, and four to five years. Interested 
readers can access the report in its entirety online at: http://www.nmc.org/publications/horizon-report-201 1-k- 12- 
edition. The present study was conducted to investigate the responses of teacher participants to the following 
questions: 
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Question 1 : Which technology in the one year or less OR two to three year categories of the Horizon Report would 
you most like to see used in your school? Why? 

Question 2: Which technology in the one year or less OR two to three year categories of the Horizon Report do you 
see least likely to be used in your school? Why? That is, what barriers exist to integrating this technology? 

The categories of one year or less and two to three years were selected as delimitations for the present study because 
it was believed that the technologies in those categories were sufficiently mature for the participants to consider as 
realistic possibilities in Education. 


Methods 


Participants 

Participants (8 male, 12 female) in this study were teachers living in Georgia who were enrolled during the spring 
2012 semester in an online, graduate-level Instructional Technology course at a public university in the southeastern 
United States. Twenty-three students elected to participate in the study, however 20 were used in the data analysis. 
Two participants were removed because they worked at the college level and the questions being investigated 
required a K-12 school context. A third participant provided incomplete responses to the discussion prompts and 
was removed from data analysis. Demographic information was compiled from participant responses in the 
discussion. Some participants provided more information than others, so full demographic information is not 
available for all variables. The participants worked in a broad range of settings. Students reporting their place of 
work indicated elementary school (n = 7), middle school (n = 3), high school (n = 4), and charter school (n = 1). 
Nine students reportedly worked in rural schools; eight students worked in urban schools. A wide range of different 
content areas were represented, including mathematics (n = 3), English (n = 2), social studies (n = 1), science (n = 

1), technology (n = 2), computer science (n = 1), business (n = 1), and special education (n = 2). Nine of the 
participants reported that they were married, and seven reported being parents. 

Data Sources 

Asynchronous, threaded discussions were utilized in the course and are delivered via the institutionally supported 
learning management system. The present study was explained to each member of the class as required by the 
institution’s human subjects review board, and each member of the class agreed to be included in the study. All the 
participants engaged in a required, beginning-of-semester introduction discussion, in which they were simply asked 
to introduce themselves to their classmates. The introduction discussion, like all of the discussions in the class, took 
place asynchronously in a threaded discussion forum in the University’s institutionally supported learning 
management system. Some students chose to include multimedia presentations with their introductions, but the 
majority of participants provided text-based introductions. These introductions were used to compile the participant 
description provided in this report. Additionally, as part of the course, the participants engaged in an online 
discussion about aspects of the New Media Consortium’s 2011 K-12 version of the Horizon Report (New Media 
Consortium, 2011). Specifically, participants were asked to do the following: 

Read the most current Horizon Report that makes sense for your professional context. Post your answers to 

the following questions in the Horizon Report discussion forum. 


a. Which technology in the one-year or less OR two to three year categories of the Horizon Report 
would you most like to see used in your school? Why? 
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b. Which technology in the one-year or less OR two to three year categories of the Horizon Report 
do you see least likely to be used in your school? Why? That is, what barriers exist to integrating 
this technology? 

c. Notes - You may not cite lack of financial resources as a reason that a technology will not be 
adopted. This reason would simply be too easy in our current economic climate. Assume for the 
purposes of this assignment that your school could afford the technologies listed, at least on small 
scale. Also, organize your responses so it is easy to see how you answered the three questions for 
this assignment. 

The participants’ responses to these prompts were used as the data sources to answer the research questions. Initial 
responses to a) were used as the data sources for question 1 and initial responses to b) were used as the data sources 
for question 2. The discussions resulting from the initial responses to these prompts were not analyzed for this 
report. The participants were instructed that financial constraints would not be accepted as reasons for non-adoption 
of the technologies, due to the economic climate at the time the study was conducted. The researchers believed that 
this restriction would eliminate an obvious and non-interesting response from the participants. 

Analysis 

Miles and Huberman’s (1994) overall strategy of data reduction; data display; conclusion drawing and verification 
for qualitative analysis was utilized for this study. Specifically, content analysis, described by Merriam (2009) as 
“the frequency and variety of messages, the number of times a certain phrase or speech pattern is used” (p. 205) was 
used for data reduction. The researchers independently evaluated the participants’ responses to writing prompt a) to 
address question 1 and generated a list of the technologies most desired to be adopted (see Table 1). Similarly, the 
researchers employed content analysis to the data collected from writing prompt b, to address question 2 and 
generated Table 2 showing the frequencies of the technologies identified as least likely to be adopted. The 
researchers' independent analysis, and resulting consensus, increases the reliability of the results. 

Results and Discussion 

The most desired technology was mobiles. Mobiles are “a category that defies long-term definitions” (New Media 
Consortium, 2011, p. 6), but the Horizon Report includes smartphones and tablet computers in its discussion of 
mobiles. The technology perceived as least likely to be adopted in schools was game-based learning. The Horizon 
Report discusses digital games for learning with examples from education publishers, commercial games, games for 
mobile devices, online games, and games for specific game consoles. The frequency counts observed for both 
desired and least likely to be adopted technologies indicate a great deal of agreement among the participants. Next, 
the reasons for their selections will be examined. 


Table 1. Frequency of choices of desired technologies 


Technology 

Number of Participants Identifying 

Time to adoption 

Cloud computing 

5 

1 year or less 

Mobiles 

11 

1 year or less 

Game-based learning 

3 

2 to 3 years 

Open content 

1 

2 to 3 years 
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Table 2. Frequency of choices of technologies least likely to be adopted* 


Technology 

Number of Participants Identifying 

Time to adoption 

Cloud computing 

2 

1 year or less 

Mobiles 

5 

1 year or less 

Game-based learning 

10 

2 to 3 years 

Open content 

2 

2 to 3 years 


*The answer of one participant was not applicable to the question. 


Why they chose the most desired technology 

Participants gave a wide variety of reasons for choosing mobiles as the technology they would most like to see used 
in their schools. Some reasons are related to students’ abilities and enjoyment. Other reasons are educationally 
focused. Still other reasons focus on the aspects of the devices. 

Several participants agree that an advantage of mobile devices is that they are engaging. “Students love them,” 
according to a first grade teacher in a rural elementary school. Students are enthusiastic about the use of technology 
and comfortable using it. “It would be a great idea to take advantage of [students’] enthusiasm for technology use 
and find great ways to integrate [mobiles] into the classroom,” wrote one female teacher. Students are mobile- 
driven, and many of them already own mobile devices and know how to use them. Participants reported that 
students need to know how to use mobile devices to prepare them for the future of continually changing technology. 
Students need to know how to access the Internet and how to create authentic products. There are many functions of 
mobile devices that students can use in educational settings. They can use them to type a paper. According to a 
female teacher of a rural elementary school, “students can better express themselves creatively with advanced video, 
audio, and imaging resources.” Students can have the ability to research any topic at any time. They could have 
access to networked information. Students can even use mobile devices to complete projects outside of school. 

Participants identified many educational benefits of using mobile devices. The use of mobile devices is suitable for 
all subjects in school. Mobile devices can make learning fun for the students. They can help students excel in 
learning, such as by using remediation software to combat low test scores. Teachers can increase the amount of 
learning and interactivity with technology when they use mobile devices. The use of mobile devices can allow for 
better retention and relevance of the subject matter. It enables teachers to incorporate new technology standards in 
the classroom. A special education teacher of an elementary school said that mobiles can be used “as assistive 
technology tools for our special needs students as well.” They can accommodate individual learners’ needs with 
easily adjustable font size. A male teacher who works in an urban high school suggested that if electronic books 
could be used through the mobile devices, hard-copy textbooks will not be needed. The use of e-books could reduce 
costs for a school system. Students could have better learning opportunities when they have constant access to their 
books. Mobile devices allow education to be taken anywhere. 

Many components of mobile devices were described as good reasons to use them in the schools. Mobile devices 
have many different uses and are easy to use and share. Ease of mobility is obviously an advantage of mobile 
devices. Mobile devices can also be considered easier to maintain than a traditional computer. A male teacher at an 
urban elementary school said that mobiles “have so much power and potential, which makes the possibilities for 
their use literally endless. The wide variety of educational [applications] make them an excellent teaching and 
learning tool.” Several other teachers also listed the variety of educational applications as a reason they would like 
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to see mobiles used in their schools. Another reason given for using mobile devices in schools is to keep up with 
technology, because it is not going away. The convenience of e-books, a variety of other tools, access to 
information, and social networking adds to the advantages. The comments of the teachers are summarized nicely 
with text from the Horizon Report: “The portability of mobile devices and their ability to connect to the Internet 
almost anywhere makes them ideal as a store of reference materials and learning experiences, as well as general-use 
tools for fieldwork, where they can be used to record observations via voice, text, or multimedia, and access 
reference sources in real time” (New Media Consortium, 2011, p. 15). 

Why they chose the technology least likely to be adopted 

Participants gave a variety of reasons for choosing Game-based learning as the technology that is least likely to be 
used in their schools. Many of the reasons given involved policies and the administration of the school systems. 
Others were due to teacher and parent beliefs. A few participants addressed concrete issues related to game-based 
learning. 

Many of the reasons given for the technology of game-based learning to be least likely to be used in schools are 
based on the administration, particularly its beliefs and attitudes. “Gaming has a negative connotation associated 
with it from the educator’s point of view,” reported a female teacher of a large urban high school. Participants 
believed that the “school administration has a negative attitude toward game-based learning.” One participant 
commented that school administrators might not think game-based learning is appropriate or view it as effective. 

An elementary school teacher commented, “our administration does not agree with playing games in the classroom 
after kindergarten.” General comments about issues related to school administrations being resistant to game-based 
learning were common. Parents and teachers also have negative opinions about game-based learning. According to 
a female teacher of an urban elementary school, “many parents and even teachers believe that students cannot learn 
through play, and [play] is a waste of time.” Parents do not like the idea of sending their children to school to play. 
Certain teachers cannot accept game -based learning as an instructional method; they believe that it is not an efficient 
way to present information. A female teacher of a rural elementary school does not think that “teachers are at a 
point where they can accept game-based learning and implement it in an efficient manner.” Other participants 
believed that students will just press buttons on games competitively without learning the material or testing their 
knowledge. 

Participants also detailed several perceived problems that lie within game-based learning. At this point, participants 
stated that there is a lack of prepared material that is relevant to the content. There is a lack of knowledge about the 
tools used to create relevant material as well. One male teacher said, “gaming designers are spending more money 
on war simulations as opposed to learning simulations. The [educational] games that are available lack the realism 
and attention to detail that is found in consumer games.” 

Because the participants were nearly evenly divided in terms of urban vs. rural schools, the researchers were curious 
if there would be a difference in teachers’ perceptions between the contexts. Cultural and financial differences 
between rural and urban and schools were suspected as possible causes in the differences. The small number of 
responses makes a statistical interpretation suspect, but the evidence displayed in Tables 3 and 4 does not show an 
obvious difference between rural and urban schools. 


112 



Table 3: Technology teachers would most like to see used in their schools: Rural vs. Urban schools 


Technology 

Rural 

Urban 

Cloud Computing 

2 

2 

Mobiles 

6 

4 

Game-based Learning 

1 

2 

Open Concept 

0 

0 


Table 4: Technology that is least likely to be used in their schools: Rural vs. Urban schools 


Technology 

Rural 

Urban 

Cloud Computing 

0 

1 

Mobiles 

1 

2 

Game-based Learning 

6 

4 

Open Concept 

1 

1 


Conclusions 

The research conducted for this study has provided some information regarding teacher beliefs about the adoption of 
emerging technologies in K-12 public schools. Understanding these beliefs can be an important step in planning for 
the successful use of technologies in the classroom. Based on the findings of this study, it appears as though 
teachers are ready to embrace mobile technologies in their practice. They stated several reasons that they do not 
believe game-based learning will be an addition to their classes in the near future. Anyone with the goal of 
integrating game-based learning in public schools may find the information provided by this research helpful in their 
planning. 

This research study is not without limitations. The study was conducted with a group of teachers in a 
relatively short window of time who were all located in one particular geographic region. Their responses were 
obviously influenced by what they had read in the Horizon Report (New Media Consortium, 2011) for the study, as 
several of their comments and examples were similar to those given in the report. Note that mobiles is in the 1 year 
or less category and game-based is in the 2 to 3 year window so perhaps teachers were better able to imagine the 
nearer term technologies as possible. Future research should consider these factors as possibly influential to the 
findings reported for this study. The observation that teachers in rural and urban schools may have similar beliefs 
about these emerging technologies is interesting and may warrant additional investigation. 
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Abstract 

This study empirically tests the relation between the instructional design elements and the overall meaningful 
interactions among online students. Eighteen online graduate Business courses are analyzed using bivariate and 
multivariate analysis techniques. Findings suggest that the level of overall meaningful interaction among learners 
can be improved by dividing the students into smaller groups, using introduction sections, and limited participation 
from the instructor. We also find that grades assigned to discussions and the use of synchronous communication 
tools does not improve the interactive quality of an online business course. The results of this study can prove to be 
useful for practitioners, designers and instructors, who are designing, developing or teaching online Business 
courses. 

Keywords: Online Business Courses, Instructional Design, Interaction, Discussion Threads 

Introduction 

Business education is going through a period of incredible transition. We face a wide range of trends that 
require leaders in higher education to step back and rethink how to approach the marketplace ( Acito, McDougall, 
Smith, 2008). A number of researchers have been speculating on how various forces will shape the future of 
business education (e.g., Friga, Bettis, and Sullivan, 2003; Hawawini, 2005). 

The field of distance education has a very long history; however, significant to this study, is the brief 
history of online learning and the need to expand the research base that exists specific to online pedagogy. The 
majority of research has focused on the continuous debate of comparing online courses with traditional courses 
(Strachota, 2003). The majority of such research has arrived at the conclusion that both the environment of the face- 
to-face course as well as the online course are considered to be equally as effective (Palloff and Pratt, 2001; Phipps 
and Merisotis, 1999; Saba, 2000). Bucy (2003) noted the volume of research that focuses on comparisons between 
traditional and online courses and concluded: 
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“Rather than using research to help replicate what is done in the traditional class room, 
researchers should focus on identifying what is done well in the online learning environment. 

Research should determine whether they (the students) are learning what we intended them to 
learn— NOT whether they are learning the same as in traditional methods.” (p. 7) 

Objective judgment of the student's knowledge and learning is still important (Valenti, Cucchiarelli, and 
Panti, 2001). However, the evaluation of online learning needs to go beyond these measures and consider the quality 
of the learning experience as a whole. Measures of student engagement offer such an evaluation (Robinson and 
Hullinger, 2008). 

Various techniques and teaching approaches for faculty to use when teaching online courses have been 
proposed in the literature (see, Levitt and Adelman, 2010; Sager, Azzopardi, and Cross, 2008; Flanagan and 
McCausland, 2007; and Melrose, 2006), that can help with the above mentioned issues but there is a scarcity of 
research that proposes methods to motivate students to actively participate in an online classroom (Jain and Jain, 
2010). Further. Most of the research on teaching techniques for online teaching has been done for Education or 
Nursing courses which may not be applicable to Business courses. Jain, Jain and Jain (201 1) analyzed the 
interactions for online graduate courses offered in four different disciplines and concluded that the level of 
interaction varies significantly across disciplines and future research must focus explaining the overall interactions 
within a discipline. Also, the literature lacks discussions about graduate -level teaching methods in the online 
environment (Candela, Crave, Diaz, Edmunds, Talusan and Tarrant, 2009). This study fills these gaps in the 
literature and proposes several tested techniques for improving the interactive quality of a graduate-level web-based 
business course. 


Literature Review 

John Dewey (1916) noted that “every expansive era in the history of mankind has coincided with operation 
of factors which have tended to eliminate distance between peoples and classes previously hemmed off from one 
another” (Dewey, 1916, p.lOO). Distance educators also follow this tradition of using the technologically 
“expansive” of eras to remove the distances from distance education. 

Technology has revolutionalized the field of distance education and a majority of Universities are 
transitioning to some online delivery of the content. Growing acceptance of online delivery methods is not only a 
source of new opportunities, but also unique challenges. Currently, over 95% of “major universities” offer some 
form of online degree programs ( Acito, McDougall, Smith, 2008). From a modest beginning during the 1990s, 
online business education has grown dramatically. The website geteducated.com has profiled more than 170 online 
programs in the United States that offer graduate business degrees (Acito, McDougall, Smith, 2008). Business Week 
Online (8/18/2005) reported that the University of Phoenix enrolls over 16,000 online MBA students. Perhaps more 
significant is the growing acceptance of online education among corporate managers, the ultimate consumers of the 
product of higher education in business (Acito, McDougall, Smith, 2008). Allen and Seaman (2005) find that almost 
80% of managers now rate distance learning as “just as good” as in-resident programs. This ratio increases to 90% 
when the name of the school is associated with a strong in-residence program. 

The amalgamation of knowledge and technology permits higher education to provide learning anytime, 
anyplace, and to anyone (Aggarwal and Bento, 2000; Maeroff, 2003; Pittinsky, 2003). The online delivery mode 
provides increased curriculum flexibility and allows universities to reach sectors of the market that would otherwise 
be unavailable to a school located in a traditional college community (Acito, McDougall, Smith, 2008). This is 
especially important for working individuals and nontraditional students who are physically separated from 
campuses or cannot frequently commute to campuses. Given the recent bad economic condition, career 
advancement is often contingent on advanced degrees. However, the busy schedule and other personal 
responsibilities of a non-traditional student can be major obstacles to the pursuit of an advanced degree in a 
traditional setting. This situation has led to the growing popularity of online distance education, with its flexibility in 
terms of time and place (Carter, 2008). Using new technological and pedagogical advances, online learning is 
ideally suited to account for individuals’ particular strengths, talents, and needs (Jeffries, 2005). 

Online learning also has the benefit of transferring the control to students (Kochtanek and Hein, 2000; Lin 
and Hsieh, 2001). Students can move back and forth between Web pages, spend as much time as necessary on a 
certain topic, and revisit pages for difficult topics. With the opportunities created by this new learning technology 
also come myriads of problems. There have been number of studies that present the various problems faced by both, 
the faculty and students in the online teaching and learning environment. Students may also have isolation problems 
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and technical difficulties in an online class (Palloff and Pratt, 1999; Sweeney and Ingram, 2001). Phillips (2005) 
cautioned about insufficient teacher- students interactions and student- student interactions. 

Educational researchers have demonstrated that a successful Web-based course depends on both course 
design (Huang, 2000) and student interaction and collaboration (Stacey, 1999). John Dewey’s (1938) “transactional” 
conception activity-based education views an educational experience as a “transaction taking place between an 
individual and what, at the time, constitutes his environment” (p.43). Dewey’s description not only fits neatly with 
the complex shifting of time and place that defines distance education but also emphasizes the importance of 
interaction with the various human and nonhuman actors that constitute the environment. For Dewey (1916), 
interaction is the defining component of the educational process that occurs when the student transforms the inert 
information passed to him/her from another and constructs it into knowledge with personal application and value 
(Anderson, 2003). 

Today, however, students expect more control over when, where, how, and how fast they learn, which 
motivates faculty to change their methods of instruction and interaction (Barone, 2003). Accrediting bodies also 
recognize the necessity for interaction in higher education. According to the Association to Advance Collegiate 
Schools of Business (AACSB) International (2006), an accreditor of business schools, “participants and their 
interactions are at the center of much of what defines quality for higher education in business. Therefore, seeing that 
the proper processes are in place to secure and manage participant resources constitutes a key evaluation in assessing 
educational quality. The participants in a degree program (students, faculty members, staff, and administrators) are 
all part of a learning community playing out interacting roles in the educational process. This is true in traditional 
educational arrangements with face-to-face interactions on an institutional campus, and it is equally true in more 
recent, technology-mediated education where some, or all, of the interactions take place electronically” (p. 33). 

Interaction is a complex and multifaceted concept in all forms of education. Traditionally, interaction 
focused on classroom-based dialogue between students and teachers (Anderson, 2003). The concept has been 
expanded to include mediated synchronous discussion at distance (audio- and video- conferencing); asynchronous 
form of simulated dialogue (computer conferencing and voice mail); such as Holmberg's (1989) “guided didactic 
conversation” and mediated asynchronous dialogue (computer conferencing and voice mail); and responses and 
feedback from inanimate objects and devices, such as interactive programs and interactive television (Anderson, 
2003). 

Distance education theorists (Garrison, 1991, 2000; Holmberg, 1991; Moore and Kearsley, 1996) and 
researchers (Anderson and Garrison, 1995; Harasim, 1990; Henri and Rigault, 1996; Katz, 2000; Saba and Shearer, 
1994; Winn, 1999) have each focused on pedagogical, motivational and economic costs and benefits of interaction 
and generally ascribe critical importance to it. Both students and faculty typically report increased satisfaction and 
learning in online courses depending on the quantity of interactions (Dziuban and Moskal, 2001; Gunawardena and 
Zittle, 1997; Kanuka and Anderson, 1998). Rust (2006) in her study, found that there is a significantly positive 
relationship between the number of postings per person and the student retention rate. 

It can be seen that interaction fulfills many critical functions in the educational process. However, it is also 
becoming more apparent that there are many types of interaction and indeed many actors involved. As a result of 
this complexity, a number of distance education theorists have broken the concept of interaction down into 
component types based largely on the roles of the human and inanimate actors involved (Anderson, 2003). The 
different types of interactions found in online instruction (Davidson-Shivers and Rasmussen, 2006; Moore, 1993; 
Wagner, 2001) include the following: (a) Learner-Learner interaction which occurs when students work or 
communicate with each other in small or large groups or on an individual basis; (b) Learner-Content interaction 
which means students are working with the instructional materials or activities, (c) Learner-Course management 
system (CMS) interaction which allows students to navigate through the online instruction, complete and submit 
assignments, and track their progress and grades; and (d) Instructor-Learner interaction which occurs when the 
instructor and students work and communicate with each other (Davidson-Shivers, 2009). Kearsley (1998) claims 
that the "single most important element of successful online education is interaction among participants." He further 
states that it is "the instructor's role as a facilitator to ensure that a high-level of interaction occurs in an online 
course" (p. 3). 

Of concern to the practice of online learning is the scarcity of research, studying the impact of effective 
design of instruction on appropriate and meaningful interactions. Too much of our practice in distance education is 
not evidence based and our actions and instructional designs are often grounded on untested assumptions about the 
value of the modes of interaction (or lack thereof). Thus, the research that focuses on interaction in all its forms is 
critically important (Anderson, 2003). This study is among the first that empirical analyses the actual online 
classroom interactions to test the predication of the survey based research and theoretical models. 
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Data and Methodology 


The data for this study come from graduate level online Business courses offered at the University of 
Wyoming during the 2008-2009 academic year. Eighteen graduate level online classes, 9 from each of the regular 
semesters, are considered for analysis. Data are collected on the actual number of student postings to the formal 
instructional discussion board. The site that provided the archived data for the study is recognized as being one that 
offered a large number of online courses each semester. The postings included all comments or questions made to 
the formal instructional discussion board by the learner addressing other leamer(s). These postings were then sub- 
classified as learner-content interactions, learner-course management system (CMS) interactions, social interaction 
and parroting, based on the rules developed by the researcher (Table 1). To provide validity to the analysis, this sub- 
classification was done in consultation with the instructor of record for the course under consideration. 


Table 1. Rules for Sub-categorizing the online student discussion board posting 


Activity 


Description 


Example 


Learner- Learner 
Interactions 


Group skills: a generic code applied to 
expressions that encourage group activity 
and cohesiveness 

Organizing work: Planning group work; 
setting shared tasks and deadlines. 
Initiating activities: Setting up activities 
such as chat sessions to discuss the 
progress and organization of group work. 
Monitoring group effort: Comments about 
the group's processes and achievements. 


Help seeking: Seeking assistance from 
others. 

Feedback seeking: Seeking feedback to a 
position advanced. 

Help giving: Responding to questions and 
requests from others. 

Feedback giving: Providing feedback on 
proposals from others. 

Exchanging resources and information to 
assist other group members. 


I know that [names] have given you good 
advice, but I think it's worth knowing that 
you need patience. 

I just want to set a time-line for myself. Is 
everyone OK with that? 

I would like to chat on the blackboard. What 
about this Friday at 7.30pm SA time? 

I believe the overall contribution and 
collaboration of working as a group requires 
an increase within itself as part of our 
learning. 

Does anyone know how to read the chart on 
pg. 12 of the text..? 

What do you think about answering the 
question that. . .has put forward? 

To read the chart, look at the Appendix A of 
the text.. 

I agree with you and 1 believe. . .Good 
point... 

"With the implementation of an internet 
service . . .there has been a major shift in the 
communication function in business." 


Learner - CMS 
interaction 


Sharing knowledge: Sharing existing 
knowledge and information with others. 

Challenging others: Challenging the 
contributions of other members and seeking 
to engage in debate. 

Explaining or elaborating: Supporting one’s 
own position (possibly following a 
challenge). 


Help seeking: Seeking assistance from 
others about the use of technology. 
Feedback seeking: Seeking feedback to a 
position advanced. 


I think we also need to give thought to the 
following. 1. The issues of quality/efficiency 
in teaching and learning . . . 

I agree but I wondered about the applicability 
of the argument: "The individuals or other 
units in a system ..." (Rogers, p. 295). 

Chery, you have a good point about 
generalizations but I think the cell phone is a 
little harder to see why the less fortunate may 
need it more than the wealthy. I think it has a 
lot to do with the marketing of the 

product 

Does anyone know how to edit/add/append 
data on the student pages? 

What do you think about tutorial on how to 
in an online class? 
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Help giving: Responding to questions and 
requests from others about the use of 
technology. 

Feedback giving: Providing feedback on 
proposals from others about the use of 
technology. 

Reflecting on medium: Comments about 
the effectiveness of the medium in 
supporting group activities. 

To access the chat room, click on virtual chat 
in the blackboard; chat screen will come up; 
click on enter. . . 

I like your idea of a generic booklet and 
everyone contributing aspects of interesting 
internet services. . . 

The email for the discussion group seems to 
work OK for me. You know it has gone 
through because you actually receive your 
email back almost straight away if it has 
worked. 

Social Interaction 

Social interaction: Conversation about 
social matters those are unrelated to the 
group task. This activity helps to 'break the 
ice'. 

Regarding chat - my weekend is pretty hectic 
- 1 have my family flying in from Greece . . . 
so the Greek festivities will be in full swing. 

Parroting 

Repeat or mimic (another's words, etc) 

I agree with you 


unthinkingly, one line 
agreement/disagreement statements 

Me too... 


Adapted from Curtis and Lawson (2001) 


Actual numbers of student postings, made to the formal instructional discussion board by the learner 
addressing other learner(s), are counted for three different weeks during the semester - week 3, week 8 and week 14, 
for each of the classes included in the study. The total number of the observations for this study was 54. To control 
for the variability in class size, the count of learner postings per week was normalized by dividing by the class size. 
By analyzing the course syllabi and structure, data on exact grade weight assigned to the discussion per week, use of 
synchronous communication tools (for e.g. online chat sessions), use of introduction unit, class size and group size 
statistics were recorded. 

After dividing the overall interactions into the above mentioned sub-categories, we come up with a measure 
of meaningful interactions for each week for all the courses included in the study, which constituted the variable of 
interest for this study. 

Results 

Table 2 describes the variables included in the study and the descriptive statistics. The mean number of 
interactions per student per week was 2.97, while the standard deviation for this variable was 1.31. Learner-Content 
interaction includes all interactions among learners based on the learning tools or past experiences or any other tool 
that facilitate in learning the course content or the subject matter. The mean of number of the learner-content 
interactions per student per week was 2.74 and the standard deviation for this variable was 1.26. Learner-Course 
management system (CMS) interaction includes all interactions among learners based on the technological 
challenges faced by the learners. The mean of number of learner-CMS interactions per student per week was 0.01 
and the standard deviation for this variable was 0.03. 

Table 2. Definition of variables included in the study and descriptive statistics 


Variables 

Description 

Mean 

S.D. 

Intperweek 

Total learner interactions per student per week 

2.97 

1.31 

L-Cperweek 

Total learner-content interactions per student per week 

2.74 

1.26 

L-CMSpweek 

Total learner-CMS interactions per student per week 

0.01 

0.03 

Social 

Total social interactions per student per week 

0.09 

1.43 

Parroting 

Total parroting interactions per student per week 

0.17 

0.13 

Chat session 

If chat sessions are used or not 

0.33 

0.48 

Group Size 

Number of students in each group 

16.44 

8.17 

Class Size 

Number of Students in a given course 

24.78 

3.52 

Intro Unit 

If students were given an opportunity to introduce themselves 

0.44 

0.50 

Instructor Posts 

Instructor’s participation in discussions per week 

1.70 

3.14 

Grade weight 

Grade weight assigned to the discussion per week 

1.21% 

0.70 
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Social instruction includes all interactions among learners not related to formal content of the subject 
matter. The mean number of social interactions per student per week was 0.09 and the standard deviation for this 
variable was 1.43. Parroting includes all the one line agreement or disagreement statements or postings that repeat 
already discussed information from earlier posts. The mean of number of parroting interactions per student per week 
was 0.17 and the standard deviation for this variable was 0.13. We find that 91% of the overall interactions were 
Learner-Content interactions, 3% social interactions, 6% parroting and 0.5% were Learner-CMS interactions. 

Overall meaningful interaction and group size 

Bouton and Garth ( 1983) stated that learning is a group process: the learner actively constructs knowledge 
by formulating ideas into words, and these ideas/concepts are built upon through reaction and responses of others 
(Harasim, 1990, p. 43). A unique feature of online education is its capability to support this interactive group 
process (Gusky, 1997). The size of groups is an important element of the success of the online learning process 
(Learning team handbook, 2003). 

Most of the courses included in this study divided the students into smaller groups to facilitate learning. On 
average, the group size for the sample courses was 17 students, with the maximum group size being 27 students and 
the minimum being 5 students. Pearson's product moment correlation of -0.53 (p<0.01) suggest a significantly 
negative relationship between the group size and overall meaningful interaction. So, based on this result we can 
conclude that the smaller the group size the better is the quality and quantity of the interaction. Figure 1 suggests a 
group size of 7 students is ideal for improved student interactions. 



Number of instructors postings 

Figure 1. Relationship between the group size and overall interaction 

Overall meaningful interaction and the use of introduction section 

For meaningful learning to occur, it is very important to create a friendly environment in which the students 
feel comfortable sharing their opinion about the content. This is especially important in an online environment, 
where feeling of isolation is a major issue (Lapadat, 2002). To encourage effective discussion and learner 
participation, it is important to build a friendly and social environment in which learners feel comfortable (Collins 
and Berge, 1996; Rohfeld and Fliemstra, 1995). This can be accomplished by including an introduction section that 
puts faces on the participants, which generates considerable discussion and sparks new interest (Rohfeld and 
Fliemstra, 1995). 

Table 2 reports that about 44% of the sample courses had an introduction section. An independent samples 
t test was conducted to evaluate whether the use of introduction unit impacts the overall meaningful interaction 
among the students. The difference between the mean number of interactions was significant at 5% level of 
significance, t(52) = 3.93, p < 0.01, which suggested that use of introduction section positively impact the overall 
meaningful interactions among students. 

Overall meaningful interaction and the instructor participation 

The most important role of instructor in an online class is to facilitate student participation and learning. 
These instructor interactions are considered fundamental to online instruction (Davidson-Shivers, 2009). But the 
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instructor needs to be careful as the online instruction requires a balance of instructor involvement in discussion: not 
too much or too little (Bonk, 2004; Davidson-Shivers and Rasmussen, 2006; Davidson-Shivers, 2009). Poole (2000) 
claimed that students become more involved and responsible for their participation when the entire discussion is not 
instructor facilitated (Hew and Cheung, 2008). 

On average, the number of instructor postings for the sample courses were 2 per week, with the maximum 
number of postings being 13 per week and the minimum being zero per week. Pearson’s Product moment 
correlation coefficient of 0.47 (p<0.01) suggest a positive relationship between the number of instructor postings 
and overall meaningful interaction. Figure 2 suggests that instructors should restrict their participation in the online 
discussion forums to 10 postings per week for improved student interactions. 



Figure 2. Relationship between the level of Instructor participation and overall interaction 

Overall meaningful interaction and Social interaction 

As Internet-based education programs expand, educators are being challenged to go beyond delivering 
information to remote learners to building community among them (Dede, 1996; Harasim, Hiltz, Teles and Turoff, 
1995; Kaye, 1995; Renniger and Shumar, 2000). Several researchers have found that the social aspects of the online 
learning environment are very important (Meyer, 2003). Giving freedom to the online students to socialize with one 
another can help create the learner community. This friendly environment can impact the student learning and 
satisfaction. The strong interpersonal ties shared by community members increase the willingness to share 
information and resources, setting the stage for collaborative learning (Haythornthwaite, 2000). 

From figure 1 we observe that social interaction accounted for about 3.23% of overall interaction. Social 
interaction ranged from a maximum of 29% to a minimum of 0% in the courses included in the study. We failed to 
find any significant relationship between the social interaction and overall meaningful interaction (Pearson’s product 
moment correlation of 0.23, p>0.05). 

Overall meaningful interaction and the use of synchronous communication 

Synchronous conferences, familiar to the recreational Internet user as "chat rooms," "chat lines," and 
instant messaging systems (Rintel, Mulholland and Pittam, 2001), also have been used in various ways in online 
courses (Murphy and Collins, 1997). Synchronous conferences have the advantages of open access to the floor, 
interactivity, immediacy, and retention of a written record (Schallert et al., 1999; Lapadat, 2002). However, they 
suffer the same constraints of linearity and real-time capacity limits as the medium of spoken discussion, which is 
further exacerbated by the brevity and ambiguity consequent of the slowness of typing, and potential discursive 
incoherence due to the multiple simultaneous conversational threads (Lapadat, 2002). 

From table 2 we observe that about 33% of the courses included in the study used real time communication 
tools, such as online chat sessions. An independent samples t test was conducted to evaluate whether the use of chat 
sessions impacts the overall meaningful interaction among the students. The difference between the mean number of 
interactions was not significant at 5% level of significance, t(52) = -1.23, p = 0.22, which suggested that use of chat 
sessions do not impact the overall meaningful interactions among students. 
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Overall meaningful interaction and grade weight 

Web-based courses included in this study incorporate an online-discussion element in their structure with a 
varying grade weight assigned to it. It is not unusual for instructors to encourage, and in some cases require a certain 
amount of participation in the form of postings per week in online discussions as part of the grade for the course 
(Sener, 2001). The research literature on Web-based learning supports this approach (Picciano, 2002). A research 
study by Jiang and Ting (2000) found that grades for discussion and requirements for discussion were significantly 
and positively correlated to students' perceived learning. Rovai (2003) conducted an ex post facto study of 18 
graduate-level courses and found evidence that grading strategies significantly influenced online discussions and 
discussions were related to students’ sense of community. In particular, he found that there were significantly more 
discussions per student per week and higher levels of sense of community in courses where discussions were a 
graded course component (Rovai, 2004). However, if the students have high level of intrinsic motivation, maybe, 
they do not require a disciplining tool for active participation (Jain and Jain, 201 1). 

The average grade assigned for discussions per week for the sample courses was 1.21% with maximum 
grade assigned being 3% and minimum being 0%. This study failed to find any significant relationship between the 
grade assigned to discussions and overall meaningful interaction (Pearson’s product moment correlation = 0.02, 
p>0.05). 

Regression Analysis 

Multiple regression analyses were used to identify the individual contribution of each of the independent 
variable included in the study. 


Table 3. Summary of regression analyses results. 


Variable 

P 

Standardized (3 

t-statistic 

p-value 

VIF 

Intercept 

1.98* 

- 

5.26 

>0.001 

- 

Group Size 

-0.14* 

-0.90 

-6.81 

>0.001 

3.11 

Chat session 

-0.36 

-0.13 

-0.89 

0.376 

3.74 

Intro Unit 

0.32* 

0.24 

2.22 

0.032 

2.34 

Instructor Posts 

0.15* 

0.35 

2.48 

0.017 

3.68 

Grade Weight 

0.07 

0.03 

0.26 

0.794 

2.69 

Social Interaction 

-0.44 

-0.05 

-0.56 

0.578 

1.31 

Instructor effect 

Yes 

- 

- 

- 

- 

Semester Effect 

0.03 

0.01 

0.10 

0.922 

2.15 


* Significant at 5% level of significance 


After testing the assumptions of linearity, homogeneity and noncollinearity, a multiple regression analysis 
was conducted to evaluate how well the independent variables predicted the overall meaningful interactions per 
student per week. To avoid the potential omission bias, we also controlled for the instructor, course and semester 
specific effects. Table 3 summarizes the result of the regression analysis. The linear combination of independent 
variables was significantly related to overall meaningful interactions among learners, F (8, 45) = 16.88, p < 0.01. 
The coefficient of determination ( R 2 ) was 0.75, indicating that 75% of the variance of the overall meaningful 
interactions in the sample can be accounted for by the linear combination of the independent variables included in 
the study. The results of the regression analysis showed that group size, use of introduction unit and number of 
instructor postings were significant predictors of the dependent variable at 5% level of significance, while the use of 
chat session, level of social interaction and grade weight were not significant in predicting the overall meaningful 
interactions per student per week. If we look at the standardized coefficient, we find that group size is the most 
important predictor of the overall meaningful interactions. 

Discussion and Conclusion 

E-learning is an increasingly common form of instructional delivery available in academic and business 
arenas (Halawi, Pires and McCarthy, 2009). Institutions are rapidly expanding the online course offerings to serve 
the nearly 4 million students and most of these institutions are new to online delivery of education (Allen and 
Seaman, 2008). With the increased pressure on online business programs, the quality of these programs is often 
questioned. 

Literature review has confirmed that little is known about these programs and their curricular foundations. 
Specifically there is need to understand how a quality online business program can be designed. Answering this 
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question is critically important, not only for evaluating the currently offered courses, but also to arrive at best 
practices to guide the future development of distance education programs in business. 

AACSB (2006) asserts that a distinctive feature of higher education is substantive and substantial student- 
student and instructor-student interaction and that “the most effective learning is highly interactive, and schools are 
expected to show that such interactions take place as a normal part of the learning experience of students in degree 
programs (p. 56). Thus, the research that focuses on interaction in all its forms is critically important (Anderson, 
2003). This study is the first quantitative analysis of online graduate business online courses offered at a major land 
grant university. The results from this study are directly applicable and might prove to be useful for developing high 
quality and interactive online business courses. Following conclusions can be drawn based on the results of this 
study: 

• A unique feature of online education is its capability to support the interactive group process (Gusky, 1997). 
Literature suggests that dividing students into smaller groups facilitate learning. The results from this study 
support this view, but the course developer should be careful to not to divide students in too small of a group. 

• Literature suggests that using an introduction section can help create a learning community, which, in turn, can 
improve the interactivity of an online classroom. This study finds support for this notion. 

• The most important role of instructor in an online class is to facilitate student participation and learning. 
However, the instructor needs to be careful as the online instruction requires a balance of instructor involvement 
in discussion: not too much or too little (Davidson-Shivers, 2009). This study finds support for the literature. 

We find that the relationship between instructor participation and overall meaningful interactions among 
learners is non-linear. Hence, to improve the interactive quality of an online class, a proper balance of instructor 
participation in discussion boards is required. 

• The social interaction highlights the importance of community for online students (Conrad, 2005; Lock, 2002; 
O'Reilly and Newton, 2002). Conrad (2005) defined community as "a sense of connection, belonging, and 
comfort that develop over time among members of a group who share a purpose or commitment to a common 
goal" (p. 2). This study fails to find any statistically significant relationship between social interaction and 
active discussion among students. 

• Literature supports the idea of grading the discussion participation to motivate students to actively participate 
online discussion forums. This study fails to find any support for the literature. The courses selected in this 
study were graduate level courses and hence, consisted of mostly non-traditional students. These students might 
be intrinsically motivated and may not require any disciplining tool for active participation. 

• Synchronous conferences have the advantages of open access to the floor, interactivity, immediacy, and 
retention of a written record (Lapadat, 2002). But, using synchronous communication tool takes the flexibility 
off of the online course. This study fails to find any significant relationship between use of synchronous 
communication tools and interactivity in an online classroom. As mentioned earlier, the sample classes 
consisted of mostly non-traditional students, who have lots of other responsibilities and time limitation and 
hence, they might prefer online courses for the flexibility provided by this mode of delivery of content. Using 
synchronous communication takes that flexibility off and may not be preferable by these non-traditional 
students. 
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Abstract 

Data gathered online from 147 adult social media users were used to validate the eight-item, Likert-type, Social 
Media Learning (SML) scale. Instrument validation procedures included analysis for internal consistency reliability, 
principal components exploratory factor analysis, and multidimensional scaling. Cronbach's Alpha for all 8 items of 
the SML scale was found to be “respectable” (Alpha = 0.78) according to reliability guidelines by DeVellis (1991). 
Exploratory factor analysis indicated two factors existed among the eight items: Interactive Learning (Alpha=0.65) 
and Learning Communications (Alpha=0.73). This research indicated that the Social Media Learning scale is 
worthy of further use in the measurement of student preferences in social media, such as Twitter, Facebook, 
Google+, and other similar tools in course communications. Accurate measurement of concepts such as these could 
support new models for interactive teaching and learning within the Web 2.0 communications environment of the 
21st century. 

Key words: social media learning, technology attitudes 

Introduction 

The Social Media Learning (SML) scale was developed from a larger student survey used to measure 
student perceptions of Twitter for course interaction as well as reflections in undergraduate courses in global policy 
and digital textuality at a Texas university. The student survey was analyzed and refined by college faculty and 
learning technologies graduate students. One section of the survey, initially referred to as the Twitter scale, was 
found to be of great interest to students and faculty alike due to the rising role of social media for communications in 
academia together with the interesting psychometric scale properties identified: the Twitter scale came to be called 
the Social Media Learning (SML) scale. The refinement process revealed that two subscales: SM Learning 
Communications (Alpha = .73) and Interactive Learning (Alpha = .65) having respectable and minimally acceptable 
reliability, respectively (DeVellis, 1991). Cronbach's Alpha for all 8 items of the SML scale is respectable (Alpha = 
.78), according to internal consistency reliability guidelines provided by DeVellis (1991). 

The courses where Twitter had been implemented as a mandatory communication tool were both designed 
to combine and extend classroom discourse and to provide students with additional materials, spontaneous updates, 
and opportunities for interaction via social media. The course instructors and the instructional designer were seeking 
avenues by which students could construct their own knowledge and engage in discourse to make greater sense of 
the course materials, expanding learning via blogs and tweets to support expressions of concepts that would 
highlight central points interactively. The intent was to support student interaction with content and increase 
knowledge construction. Students were provided with prompts to invite discourse for development and display of 
concepts as well as to critique course components. The guided instruction by the instructors as well as the 
collaborative interaction between the students via social media in these university courses in many cases contributed 
to engaging discourse. Twitter messages are short, 140 characters or less. This provides learners with both a 
challenge and promotes students sharing very succinct messages. Technology discourse is becoming commonplace 
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in 21st century educational environments. It is an important component of distributed learning and teaching because 
it allows for a more active mode of communication. 


Review of Literature 

Social Media in Education 

Educators and institutional administrators are increasingly promoting and trying out social media tools in 
attempts to open up communication channels, to tie students closer to their institution (Heiberger & Harper, 2008) 
and to engage students more in their classes (Junco, Heiberger, & Loken, 2010; Junco, 2012a). Such tools include, 
for instance, the popular among students ‘Facebook’ application and the microblogging tool ‘Twitter.’ Reasons 
often given for using such tools in an educational setting include the need to meet students where they already are 
(Bodle, 201 1), especially in the online spaces they inhabit (Heiberger & Harper, 2008). Research-based evidence 
from the usefulness of social media implementations for learning purposes, however, remains limited (Hew, 2011; 
Junco, et al., 2010). There is larger body of evidence to indicate that excessive, non-instructional social media use 
cannot be grouped with educationally beneficial activities. 

The Higher Education Research Institute (HERI, 2007) reported from a large scale study that extensive 
socializing online (more than six hours) negatively impacted students study time and study habits. Later quantitative 
studies have confirmed this. Junco’ s (2012a) study that explained the negative relationship between student 
engagement and Facebook use, as well as Junco’s (2012b) and Kirschner and Karpinski’s (2010) studies that 
showed how time spent on Facebook was negatively associated with overall GPA - all indicate that too much social 
media socializing may have an impact on study-time and thereby on learning. Smith showed in a 201 1 study how 
Americans in general use social media and several studies have further suggested how educators can use social 
media in the educational landscape for community-building, social learning, and borderless collaboration (Hsu & 
Ching, 2011; Dunlap & Lowenthal, 2009; Cochrane, 2010). Recommendations include uses that promote being 
together and closeness; that is, a social presence. In an experimental mixed-methods study in a fully online course, 
however, Wakefield, Warren, and Alsobrook (201 1 ) noted that students had diverse perceptions of whether or not 
the implementation of Twitter as a mandatory communication tool contributed to a sense of social learning 
community. 

As noted above, the majority of research conducted so far relating to use of social media within education 
has been focused on engagement or social presence. Examples include Walter & Barazova’s (2008) study on how 
social media allows for propinquity within groups and addresses the perceived impact on social presence. 
Researchers have also looked at number of tweets and their communicative purposes (Boyd, Golder, & Lotan, 2010; 
Honeycutt, & Herring, 2009) and instances of job postings over Linkedln, a social network for professionals 
analyzing qualifications needed as specified by employers (Wakefield. Warren, & Mills, 2012). 

Other than the social presence and social community studies, experimental studies using social media in the 
classroom are few. Examples include Borau, Ullrich, Feng, and Shen's (2009) study where the researchers used 
Twitter successfully for language learning. Further, the ICMPA (2010) and Flippin-Wynn & Tindall (201 1) studies 
looked at how social media (or the absence thereof) now strongly impacts students’ lives. 

What are missing in the educational environment are new reliable instruments to gauge this new type of 
learning - student learning with social media - particularly in the classroom setting. Recognition of ways in which 
information, communication, technology, and learning (ICTL) are changing nearly all facets of our lives 
(Christensen & Knezek, 2008), as well as new standards for technology literacy and technology integration, have 
created a need for a broader examination of the role that 21st century technology can play in teaching and learning 
(Mayes, Mills, Christensen, & Knezek, 2012). Interest in student social learning includes examination of social 
media learning and student career choices in seeking to understand STEM career dispositions as related to how 
students prefer to use technology (Knezek, Mills, Wakefield, & Hopper, 2012). In the current research study we 
validated a prototype instrument, the Social Media Learning (SML) scale, which measures preference for learning 
with social media. 

Few quantitative research papers were found reporting on the use of scales measuring other than what 
people are doing while using social media. However, the scale items in Junco’s (2012a) study on the relationship 
between Facebook use and its impact on student engagement and Junco, Elavsky, & Heiberger's (2012) mixed- 
methods study on Twitter and student engagement both build on The National Survey of Student Engagement 
(NSSE) instrument (NSSE), a validated instrument with acceptable psychometric properties assessing student 
engagement in good educational practices and gains from the college experience (Kuh, 2001). Another such study is 
Dholakia, Bagozzi, and Pearo’s (2004) research on virtual community participation, which builds on items from 
Flanagin and Metzger (2001). Several studies where found where unvalidated instruments had been used, i.e. new 
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instruments (Hughes, Rowe, Batey, & Lee, 2012; Joinson, 2008; Mazman & Usluel, 2010). Another such study is 
Wakefield, et al.’s (201 1) study on students’ use of Twitter targeting student comfort, perceived community, social 
presence, and interactive learning in a course on Global Policy issues. The SML instrument evolved from this 
Global Policies Twitter instrument and was further developed in another university course. Subsequently the SML 
survey instrument was analyzed and further developed by Alsobrook, Wakefield, and Knezek in 201 1 into the Social 
Media learning (SML) scale. 


Instrument Development 

Instrumentation indicates rigor (Straub, 1989), and therefore we want to ensure the instrument we are using 
in our current research has undergone examination for validation. The Social Media Learning scale was developed 
to help understand students’ reactions to social media applications such as Twitter, Facebook, Google+, and similar, 
in support of undergraduate university curriculum. Instrument development included analysis for internal 
consistency reliability, principal components exploratory factor analysis, and multidimensional scaling. A review of 
the literature revealed a need for validated instruments to measure students’ preferences for social media discourse 
interactions centered around university coursework. Items for SML Version 1.0, an 8-item Likert-type instrument, 
are displayed in Table 1. Responses are rated on a 5 point scale with response choices from 1= strongly disagree to 5 
= strongly agree. 

Table 1 . The Social Media Learning Scale. 


Instructions: Select one level of agreement for each statement to indicate how you feel. 


SD = Strongly Disagree, D = Disagree, U = Undecided, A = Agree, SA = Strongly Agree 



Social Media Learning 
When using social media... 

SD 

D 

U 

A 

SA 

1. 

I feel a sense of community 


(2 1 

(3) 

© 

i5) 

2. 

learning becomes interactive 

© 

(2) 

(3) 

© 

(5,i 

3. 

posting questions to my peers helps me understand my 
readings better 


(2 - 

(3) 

© 

(5) 

4. 

I am able to get faster feedback from my peers 


d> 

3 1 

© 

(5,i 

5. 

I am able to get faster feedback from my instructor 

© 

(2) 


© 

(5,i 

6. 

I am able to communicate effectively 

© 

(2,i 

(3) 

© 

(5,i 

7. 

I am able to connect with peers more easily than face-to-face 

© 

(2i 

(3) 

© 

(5,i 

8. 

I increase my participation in classes when I am allowed to 
contribute through social media 


(2) 

(3) 

© 

(5 ' 


Note: Social Media Learning scale, SML.V.l. By Alsobrook, M., Wakefield, J. & Knezek, G., (2011). 

Data Collection 

Survey subjects were volunteer adult social media users who responded to email and facebook invitations 
to participate in a study of learning preference. One hundred forty-seven (147) respondents completed an online 
Learning Preference battery that included the SML scale during the spring of semester of 2012. The study 
participants were 76% women (n= 1 1 2) and 24% men (n=35) spanning 18 to 69 years of age. 


129 


Instrument Refinement 


Scale Construct Reliability and Validity 

Internal consistency reliability analysis was used to assess the reliability of the SML scale. As shown in 
Table 1, Cronbach’s Alpha for all 8 items of the SML scale is considered “respectable”, (alpha = .78) according to 
reliability guidelines by DeVellis (1991). Exploratory factor analysis, principal components analysis (PCA) with 
varimax rotation, was conducted in order to identify scales/factors that are orthogonally aligned (Mertler & 
Vannatta, 2005). 

Analysis of the exploratory factor analysis Scree plot (see Figure 1) indicated one or possibly two factors 
should be retained, based on the Scree plot of factors with eigenvalues greater than one and, according to Stevens' 
(1996) suggestion that constructs in the sharp decent of the graph, before the first point of leveling, should be 
retained. The two-factor solution was retained. A scale constructed from Items 1,2,7 and 8 resulted in Alpha=0.66, 
and items 3,4, 5 and 6 produced Alpha=0.68. These Alpha's were considered only minimally acceptable according 
to guidelines by DeVellis (1991) and would have resulted in two substandard scales. Further refinements are 
presented in the following sections. 


Scree Plot 



Figure 1. Scree plot for exploratory factor analysis of SML items 
Multidimensional Scaling 

Multidimensional scaling was conducted to further examine the underlying factors for the SML scale. The 
ALSCAL Euclidian distance model with maximum of two dimensions was generated to examine distances and 
proximities for items in relation to one another. As shown in Figure 2, the two main output groupings visible in the 
Euclidean distance model are separated by the vertical axis, revealing one respectable subscale (items 4,5, 6,3,1) 
and one minimally acceptable subscale (items 8,2). The first cluster/subscale has its strongest factor loading on item 
4, 1 am able to get faster feedback from my peers, and was named SM Learning Communications. The second 
subscale with strongest loading on item 8, 1 increase my participation in classes when I am allowed to contribute 
through social media, was named Interactive Learning. Upon review of the scale items, the research team agreed 
that these factors / subscales have reasonable content validity with respect to the desired research domain, and 
therefore this two factor solution was accepted. Factor analysis and multidimensional scaling confirmed a two 
construct solution which was verified by the research team to also possess face (content) validity. Items comprising 
each of the Social Media Learning subscales are listed in Table 2. 
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Table 2. Social Media Learning Items 


Social Learning Communications 


When using social media. . . 

4. 1 am able to get faster feedback from my peers. 

5. 1 am able to get faster feedback from my instructor. 

6. 1 am able to communicate effectively. 

3. Posting questions to my peers helps me understand my readings better. 

1. 1 feel a sense of community. 

Interactive Learning 


When using social media. . . 

8. 1 increase my participation in classes when I am allowed to contribute through social media. 
2. Learning becomes interactive. 


Derived Stimulus Configuration 


ai 

E 

Q 


Euclidean distance model 



Dimension 1 


Figure 2. Euclidean distance model. Multidimensional scaling plot of SML items. 

Criterion-Related Validity 

Criterion related validity was examined for the scales of SML by correlation analysis with the Integrated 
Communications Technology Learning (ICTL) scale, a 15-item, Likert-type survey instrument developed by Mills 
and Knezek (2012) for research on learning preferences in higher education. ICTL assesses student preference for 
learning with integrated communications technology options of the Web 2.0 world of the 21st century, specifically 
for information seeking, and information sharing, as shown in Figure 5. Mills, Knezek and Khaddage (2012) found 
the internal consistency reliability for the Information Seeking (Alpha = .71) and Information Sharing (Alpha = .83) 
to be respectable and very good (DeVellis, 1991) among n=62 undergraduate and graduate university students in a 
201 1 study among higher education learners (Mills, Knezek & Khaddage, 2012). The items comprising each scale 
are listed in Table 3. 
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Table 3. ICTL Scales 


Information Seeking 


13.1 learn more when I regulate my own learning experience and seek information on things that I want to learn 
about. 

2. 1 use Internet technology to explore topics of interest. 

7. 1 like to take classes from good professors. 

10. Internet technology helps me be successful in my college classes. 

4. 1 like to enroll in classes to continue my education. 

8. 1 use Internet communications technology tools when I want to leant about something new. 

14. 1 use Internet communications technology to keep current on topics related to my field of expertise. 


Information Sharing 


3. 1 like to share interests and reflections online. 

5. 1 use Internet communications and other technology tools for self-expression. 

6. 1 learn many things by interacting with other Internet users. 

1 1 . More classroom learning should include interactive communication technology experiences. 

1. 1 would like to be a participating member of an online community. 

15. 1 post information that might be of interest to other people. 

9. 1 learn best in a traditional classroom setting. ( R) 

12. The things I need to know are taught by instructors in the classroom. 


Criterion-Related Validity 

Bivariate correlation was examined for Social Media Learning and Integrated Communications 
Technology Learning scales to determine if SML scales would align as an indication of criterion related validity. 
Significant (p < .05) correlations were identified between SML scales (SM Learning Communications, Interactive 
Leaning) and ICTL scales (Information Seeking and Information Sharing). The specific correlations were: 

1) SM Learning Communications and ICTL Information Sharing r = 0.55 (p < .0005), 

2) SM Learning Communications and ICTL Information Seeking r = 0.33 (p < .0005), 

3) Interactive Learning and ICTL Information Sharing r = 0.40 (p = .009), and 

4) Interactive Learning and ICTL Information Seeking r = 0.21 (p < .0005). 

Correlations (p < .01) were also identified between the Social Media Learning (SML) scales and other 
factors measured for n=147 participants in the study: a) SML total scale score and participant gender r = 0.22 (p = 
.007), b) SM Learning Communications and participant age and r = - 0.35 (p < .0005), c) SM Learning 
Communications and home internet usage hours r = 0.32 (p < .0005). 

To summarize, participant age is negatively correlated with total preference for SML. Gender is positively 
correlated with SML, meaning females prefer social mediated learning more than males. Mean scores for SML total 
scale score for men (3.11) differed from women (3.42), with moderate Cohen's D effect size of 0.52. SML scales for 
Learning Communications and Interactive learning are both more strongly correlated with a preference for sharing 
information than for seeking information, based on ICTL scales. 

Discussion 

Aittola ( 1999) pointed out that the role of formal education as the disseminator of significant learning 
experiences has diminished. ICT tools such as those designed for social media support a variety of distributed 
learning, sharing, and knowledge construction activity options (Knezek, Lai, Khaddage & Baker, 2011) thereby 
creating new avenues for formal and informal learning interaction. Learner preference for social media 
communications within an educational context can be measured by validated instruments such as the Social Media 
Learning scale. With appropriate instructional design educators and technology developers can be assisted in 
meeting the challenges identified by Naismith, Lonsdale, Vavoula, and Sharpies (2004), mainly that of finding ways 
to ensure that new learning methods are effective - highly situated, personal, and collaborative for the long term. 
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and accepted by students. An understanding of student preferences is key to identifying students’ level of 
acceptance, opinions and expectations regarding the integration of new technologies into teaching and learning. The 
development of such an understanding is what Andrews and Tynan (2012) have referred to as “investigating the 
human voice” (p. 565) in order to better meet the needs of today’s learner. 

Summary and Conclusions 

This research validated (showed the appropriateness of ) the Social Media Learning scale for measurement 
of student preferences for use of social media, such as Twitter, Facebook, Google+, and other similar tools, in 
course communications. Valid instruments such as the SML scale can contribute new knowledge to the emerging 
models being developing within instructional design to include use of social media. The implication is that SML 
scales for measurement of student preference toward SM Learning Communications and Interactive Learning are 
important points of consideration for designing instruction that will promote student engagement and collaborative 
learning by encouraging educational discourse. The Social Media Learning scale was found to have construct and 
content validity, aligning with the ICTL survey, as anticipated. For example, students who spend more time on the 
Internet at home have higher regard for social media learning. Further research is needed to determine if SML 
Version 1.0 and its anticipated expanded version will be useful in addressing additional research questions on 
students’ attitudes towards social media course interaction and educational communication, ICT integration, and 
instructional course design. Accurate measurement of concepts such as these could support new models for 
interactive teaching and learning within the Web 2.0 communications environment of the 21st century. 
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Abstract 

This paper will present the preliminary findings of a pilot study. Participants were randomly assigned into one of 
two groups, those receiving instruction 1) with subtitles (redundant text); & 2) instruction without subtitles. A 
diagnostic test was used to categorize learners as 1) having symptoms consistent with attention deficit hyperactivity 
disorder (ADHD); or 2) non- ADHD learners. Two performance variables were considered (retention & Uansfer). 
Finally, learner cognitive load was also measured. 


Introduction 

Mayer (2009) describes multimedia instruction as including a combination of audio, words and pictures, which may 
be presented to the learner as animation, narration, visuals, and text. The use of these various forms of media 
spawned an intense debate in the 1990s (Clark, 1983; Clark 1994; Kozma, 1991; Kozma, 1994). Rather than 
considering a media-centric approach, Mayer proposed we should turn our attention to the learner and learning 
(Mayer, 1997; Moreno & Mayer, 2000). This learner-centered approach has led to the development of a wide variety 
of instructional principles for the design, and development of multimedia (Mayer, 2009; Moreno & Mayer, 2000; 
Moreno, 2006). 

Several studies have found learners perform better given multimedia that includes narration and animation, as 
opposed to animation and text (Mayer, 2009; Moreno & Mayer, 2000). This was described as the modality effect 
(Penney, 1989), or the modality principle (Mayer, 2001). Much of this work revolves around the idea that a 
reduction of on-screen text during animated multimedia, helps learners to process their instructional materials 
(Mayer & Moreno, 1998). 

However, a conflicting body of research has shown that bimodal presentation (or redundant text) during instruction 
can aid verbal recall, especially in those with learning disabilities (poor reading skills) (Montali, & Lewandowski, 
1996). Evidence supporting these bimodal presentation techniques dates back to the 1970s (Halpern & Lanz, 1974; 
Kinchla, 1974). It should be noted that these studies only describe the presentation of text with redundant narration. 
This body of research was prior to the development of modern multimedia, which often includes concurrent 
animation, or the concurrent presentation of additional visual elements. Therefore, redundant visual elements like 
text (e.g. subtitles) during an animation sequence would likely cause a learner to split their attention between the 
animated visual elements, and the redundant text (subtitles). This is known as the split attention effect (Chandler & 
Sweller, 1992). 

According to cognitive load theory, redundant visual elements (e.g. subtitles) may impede learning if those 
additional elements could be understood in isolation (Chandler & Sweller, 1991). According to Mayer’s “cognitive 
theory of multimedia learning” (Mayer, 1997; Mayer, 2009; Mayer & Moreno, 1998) learners must integrate the 
verbal and visual information that they receive. Given narration and redundant text within multimedia, a learner may 
find this amount of information overwhelming. 
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Some learners tend to have increased difficulty with multimedia, especially if it is visually demanding. For instance, 
individuals with ADHD have a decreased ability to process visuospatial information (Alderson, Rapport, Hudec, 
Sarver, & Kolfler, 2010; Sowerby, Seal, & Tripp, 201 1). Individuals with attention deficit or hyperactivity disorders 
experience a disruption in the processing of information in the visual and auditory information, affecting their ability 
to accomplish instructional objectives (Brown, 2009). Prior research has shown learners with ADHD tend to be 
inattentive while using longer non-narrated video segments or long text documents (Solomonidou et al., 2009). 
However little research has documented how well these learners are able to retain and transfer what they have 
learned, given complex multimedia. 


Method 

The sample included undergraduate (pre-service teachers) and graduate students from two large southeastern 
universities. After agreeing to participate in the study, learners were randomly assigned into one of two groups by a 
JavaScript. These groups will be presented multimedia that either includes 1) animation and narration (AN); or 2) 
animation, narration, and redundant text (subtitles)(ANT). 

The subject matter of the multimedia was hurricane formation (Lewis, 2012). Following the instruction, all 
participants were administered a questionnaire (Likert scale, multiple choice questions, and open ended questions) 
collected via a web-based form. This questionnaire includes questions about content presented, the perceived 
difficulty of the lesson, difficulty of the instruction, their ability to attend to the lesson, perception of what they 
remembered, and demographic questions. Once learners have finished the questionnaire they were thanked for their 
participation. 


Data Analysis 

A subscale of 18 questions was embedded within a larger questionnaire. These questions were taken from a 
diagnostic tool, the ADHD Current Symptoms Scale Self Report Form (Barkley & Murphy, 1998). These questions 
are often used as a diagnostic tool to document ADHD symptoms. This study intends used this tool as a means of 
categorizing learners as either having symptoms consistent with ADHD, or as non-ADHD learners. We considered 
two sub-categories of ADHD (“impulsivity'’ or “inattentive/hyperactivity”). If a learner scored a 2 or 3, on 6 of the 9 
items (within a category) indicates that individual experiences symptoms of impulsivity or inattentive/hyperactivity 
severe enough to impact their daily life. 

In addition the questionnaire included a series of questions based upon the multimedia presentation. Nine multiple 
choice questions were used as means of measuring retention. Responses to three opened were scored to measure 
transfer. 


Results 

The results provided in this paper are those of a pilot study. The number of participants for this preliminary study 
was low (n=7) of which 2 individuals scored high enough on “the ADHD Current Symptoms Scale” to be placed in 
the ADHD group. One individual who met the criteria for impulsivity, was randomly assigned into the without 
subtitle group. The other ADHD individual met the criteria for inattentive/hyperactivity and was randomly assigned 
within the group with subtitles. The remaining participants did not score high enough for placement within the 
ADHD group. Two non ADHD learners were randomly assigned to the non subtitle group and three to the subtitle 
group. 

Although the participant groups were small, trends as related to the research question can be identified. On the recall 
questions, the non ADHD participates on average scored identically the same with both groups scoring a 9.0. 
However, differences were noted between the two ADHD individuals with the with subtitles scoring better .6 
average of correct answers as compared to the one who participated in the presentation without subtitles at .5 
average of correct answers (See Figure 1). If this trend with ADHD were to be evident in a large sample, this finding 
would not be consistent with the well-documented “spilt attention effect” (Chandler & Sweller, 1992). As other 
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researchers have shown subtitles may assist individuals with ADHD, as they could be compensating for the slower 
visuospatial information processing. 


Retention 



WO Subtitles WO Subtitles Subtitles Subtitles 
WO ADHD WADHD WO ADHD ADHD 


Figure 1 . Average of Correct Answers of Recall Questions 

The split-attention effect was evident in the transfer questions for the non ADHD group with average score of .5 as 
compared with the subtitle group score of 1.0. Both of the non- ADHD groups scored better than the ADHD 
individuals (See Chart 2). Neither of the ADHD individuals attempted to answer the transfer questions. One left 
them blank and the other wrote why they were unable to answer indicating they were unable to listen long enough to 
finish the 7.5 minute instructional presentation. 


Transfer 
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Subtitles Subtitles WO ADHD ADHD 

WO ADHD ADHD 


Figure 2. Average of Correct Answers of Transfer Questions 

Discussion 

The subtitles appear to cause non- AD HD learners to experience a cognitive overload as they process animation and 
text which splits the attention. Although no significant differences were noted given the retention questions, 
difference did emerge in the transfer questions. This is consistent with Mayer’s “Cognitive theory of multimedia 
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learning” (Mayer, 2009). The opposite was noted with the ADHD participants. Perhaps the use of subtitles gave 
additional support to these learners resulting in a higher score. 

The generalizability of this study is limited, due to the low number of participants. As more participants are 
considered, it will be interesting to observe if these findings continue to be evident. The inability of the ADHD 
participants to answer the transfer questions should also be explored. Would a different combination of audio and 
visual support improve their performance? For example, would an audio-only presentation be helpful for those with 
the slower visuopatial processing? 


Conclusions 

Several studies have shown that removing redundant material can improve learner performance as it reduces the 
load impacting learners as they attempt to leant new material (Kalyuga, Chandler, & Sweller, 1998, Kalyuga, 
Chandler, & Sweller, 1999). Mayer and Moreno (2002) found learners from the general population (without ADHD) 
who used redundant visuals during animated multimedia, performed poorer than those who did not have redundant 
visual elements. Therefore it is expected that ADHD learners will perceive redundant visual instruction (e.g. 
subtitles) as more difficult, because it is visually more demanding. However, based on this preliminary data 
collected, the individual with redundant subtitles were able to retain more information from the presentation than the 
individual without the redundancy. ADHD Individuals were presented materials both with, and without redundant 
text (subtitles) but did not answer the transfer questions. The information collected to date indicates these ADHD 
individuals performed better with redundant visual information. It is hoped that continued research (with a larger 
sample) will provide guidelines for those developing multimedia for learners with and without ADHD. 
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Abstract 

This study focuses on the role of video in pre-service teacher (PST) observations of the first day of school 
and the impact of peer reviewing the videos on learning of new teaching concepts and the future application of 
learned concepts. Since PSTs do not always have the ability to observe the first day of school directly, peer videos 
were taken during the first day of school, edited, and shared to investigate the impact of viewing field videos and 
future use of teaching practices. Online surveys were used and Spearman rho correlations were examined to review 
relationships and significance between the viewing of videos, learning and future application of teaching concepts, 
and preferences for types of field observations. Results suggest that PSTs who review videos made by their peers in 
combination of their field observations are more likely to learn new teaching concepts and plan to apply them when 
becoming a teacher. After learning new teaching concepts through watching the video related to observations on the 
first day of school, the PST’s indicated the video helped them prepare for the actual face-to-face observation and are 
likely to prefer a combination of both virtual video and face-to-face field observations. 

Introduction 

This paper identifies the importance of high-quality field experience opportunities for pre-service teachers 
(PST) and the advantages and disadvantages of using technology to enhance the field component. Field placement 
experiences are intended to provide a bridge for PSTs between classroom knowledge and theories learned and the 
practice of teaching in the real world. There are several factors that restrict the PST from engaging within field 
experience opportunities vital to the development of a highly skilled teacher. One deficiency identified within the 
teacher-training program at Ashland University and majority of teacher-training programs is the lack of a “first day 
of school” experience. The first day of school experience is vital to the teaching and learning process. It is 
fundamental in building a foundation for effective learning and teaching for the rest of the school year and yet there 
is a lack of exposing PSTs to this learning environment and expectations in most teacher-training programs. 

The use of technology in field observations is increasing as technology becomes more readily available. 
Within the teacher-training program at Ashland University, we decided to expand our current practice of utilizing 
technology as a tool to meet specific needs, in this case the need to expose PSTs to the first day of school 
experience. We wanted to identify if the use of videos created by peers would provide a deeper understanding of the 
first day of school. The students reviewing the video were not be able to experience the first day of school because 
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of scheduling issues of the field observation. We wanted to explore what the impact of experiencing video field 
observations could have on the pre-service teachers' learning of new teaching concepts and the possibility of 
implementing the concept in future teaching experiences pertaining to the first day of school video observation. The 
findings revealed a statistically significant positive correlation between PSTs learning of new teaching concepts 
from watching the video and the implementation of the concepts in future teaching. 

Literature Review 

The core of any teacher preparation program is high-quality field placements with expert teachers to act as 
mentor and role models for pre-service teachers and supervisors who can observe and provide useful feedback to 
them. Some universities have used technology to enhance these experiences. Others, because of lack of resources or 
other concerns have used technology to replace live experiences. There are advantages and disadvantages to using 
technology with field experiences. The body of research on the use of technology in field experiences is scant, but 
growing, reflecting our changing definitions of education and teacher preparation. 

Hixon, E. & So, H-J (2009) identified three types of technology-enhanced field experiences. In Type I, pre- 
service teachers are placed in real classrooms and technology tools are used by them and their supervisors to 
“facilitate supervision, reflection, and/or communication” (p. 296). This is by far the most common application of 
technology in a field setting. In Type II experiences, students have vicarious experiences by remotely viewing 
teachers and students in other real classrooms. Because of synchronous types of communication, such as Skype, the 
pre-service teachers may be able to interact with the students. Type III field experiences utilize completely simulated 
environments or “virtual practicums” as field experiences. The authors note that technical difficulties and cost 
factors are major obstacles to implementing and researching this type of approach. Nonetheless, the authors argue 
that Type I and II field experiences can “increase access to quality classrooms” and “encourage pre-service teachers 
to explore new ideas in a safe environment” (p. 301). 

Researchers have identified several advantages to using technology in field experiences. The use of 
technology to provide students with experiences with diverse populations that are beyond their geographic 
limitations is often cited as an advantage. Phillion, Miller & Lehman (2005) described how their university 
employed video conferencing with a diverse classroom to provide their pre-service teachers the opportunity to 
observe and interact with teachers and students in settings far away from their own rural environment. The authors 
noted many advantages to using two-way video technology such as helping to dispel pre-service teachers’ 
misconceptions about low-income students, observing master teachers at work, understanding the relationship 
between cultural backgrounds and school, and observing the role of cultural identity in children’s education. They 
also noted that technology is only a tool in this process, not a solution. Bondy , Ross, Gallingane & Hambacher 
(2007) also found that videotaping and analyzing the practices of novice teachers in urban school settings can lead to 
a more culturally responsive classroom management style. 

Researchers have also studied how technology can enhance affective attitudes of pre-service teachers. In a 
study of 400 student teachers, Karsenti & Collin (201 1) found that watching online teaching videos positively 
affected the self-efficacy beliefs of these pre-service teachers. They were better able to feel prepared for actual 
classroom situations by viewing another teacher first. The use of technology has also been shown to have a positive 
effect upon pre-service teachers’ ability to reflect on their actions. Fox, Campbell & Hargrove (201 1) found that 
faculty can help pre-service teachers become more reflective in their practice by audio and/or videotaping their 
lessons and engaging in a collective conversation with their peers. This enables them to “extend their learning and 
develop a common understanding of their experiences” (p. 48). Hixon and So (2009) also noted the positive role 
technology can play in facilitating student reflection. 

Other researchers have noted the usefulness of teaching videos and other technology in helping pre-service 
teachers prepare for specific content area instruction. Star, Lynch & Perova (201 1) found the use of videos helped 
improve mathematic teachers’ ability to attend to significant pedagogical practices. Morrell & Schepiage (2012) 
noted that technology enhanced pre-service science teachers’ ability to analyze their instruction. 

The most intriguing research is being done on “virtual practicums.” These can be implemented in 
convenient and easy ways such as virtual field trips (VFTs). Cox & Tyng-yn (2004) studied the impact of VFT in a 
university course over a three -year period and found, if used properly, VFTs “can add a dimension to instructional 
activities that a classroom course cannot offer” (p. 123). At the extreme end is virtual schooling (VS). Mullen, 

Beilke & Brooks (2008) examined using virtual world gaming programs such as Second Life and World of Warcraft 
to create virtual classrooms. The authors state that “virtual or simulated environments hold promise as a means of re- 
visioning the field experience component of multicultural education, and thus creating the wusettlement necessary 
for the (re) construction of a teacher identity” (p. 26). They conclude that “virtual environments hold the distinct (if 
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not sole) capability to allow pre-service teachers to do exactly that by creating new identities for themselves, taking 
on the personae of others, and thereby understanding the other’s perspective” (p.27). However, not everyone is so 
enthusiastic about the implications of virtual schooling. Compton, Davis & Correia (2010) found that pre-service 
teachers have many fears and misconceptions about virtual schools including fear of elimination of more teaching 
jobs, increased academic dishonesty from students and lack of face-to-face interaction. Nonetheless, the authors 
advocate implementing VS in early field experiences to address these fears. While generally supportive of virtual 
field experiences, Hixon & So (2009) also point out their limitations. In addition to cost and the possibility of 
technical problems, they note that students report that they view VS as missed opportunities to participate in more 
authentic field experiences and that virtual experiences are limited and lack the complexity of a real classroom. 

The first day of school is an experience most pre-service teachers do not get to encounter first hand even 
though it will most likely be their first task upon graduation. Brooks (1985) has indicated that first year teachers 
struggle with implementing successful strategies and methods on the first day of school because they have not had 
enough, if any experience. Wong and Wong ( 1998) identify that the first day of school sets the tone and structure for 
the entire year and what happens on the first day of school can determine the success or failure for the rest of the 
year. Teachers can win or lose the class on the first day and can be an accurate indicator of how the rest of the year 
will go for both the teacher and students. Wilson and Wilson (2007) also note that good first day experiences result 
in greater motivation and better performance throughout the year. Research supports the need to design experiences 
and curriculum around the first day of school experience. 

Ashland University provides pre-service teachers with over 600 hours of supervised field experiences in 
real classrooms with master mentor teachers. A variety of technology tools such as teaching videos, discussion 
forums, blogs, wikis, are used to enhance student preparation. Nonetheless, technology was used to solve the “first 
day of school” issue. Candidates in Early Childhood (PreK-3rd grade licensure) who did their student teaching in the 
spring could not experience the first day of school. It was crucial for PSTs to observe this since so many things are 
accomplished on that first day in a primary classroom: rules and routines are established, young children must 
separate from their parents and learn to adapt to a new environment, and new teachers must establish rapport with 
their new classes. A solution was to have pre-service teachers in the fall semester film the first day of school at two 
elementary schools with iPads and flip cameras, with the help of education technology professors, created a DVD 
that was then shown to the PSTs in the spring semester. This article describes the results of the study comparing 
students who experienced a “real” first day of school experience with those who had a “virtual experience.” 

Methods 

A correlational study was used to investigate the relationship between the viewing of peer observational 
video from the first day of school and the learning and future use of teaching practices. A cross-sectional self- 
reporting survey was used to examine PSTs perceptions of a video of the first day of school created by their peers 
from the Fall Semester on their learning and influence on future teaching practices. Fall semester students were 
given iPads and digital video cameras to twenty-two pre-service teachers. Using these mobile devices the Fall 
semester PSTs conducted their field experience at two elementary schools within the same district during the same 
three-hour time frame. 

After viewing the field observation video. Spring semester PSTs were e-mailed a link to an online survey. 
Eleven students from the class completed the survey. The survey consisted of 14 questions and would take an 
English speaking person approximately 10 to 15 minutes to complete. Ten questions were closed ended with the use 
of Likert scales and labeled as continuous variables. Four questions were open-ended. The self-constructed survey 
was analyzed for validity by 2 experts in the field reviewing for content-related evidence of validity. 

Percentages were calculated for each question. Spearman Rank Order correlations were used and examined 
to review relationships and significance between the viewing of videos, learning and future application of teaching 
concepts, and preferences for types of field observations (face-to-face vs. virtual video). Spearman’s rho 
correlations were used due to the Likert scale responses being categorical (Field, 2009). The non- statistical and 
statistical assumptions were tenable. The open-ended questions were also analyzed to provide a more insight into 
PST thinking and use of video within their observations. 

Findings 

The Spearman's rho revealed a two-tailed statistically significant positive correlation between PSTs 
learning of new teaching concepts from watching the video and the implementation of the concepts in future 
teaching rs( 1 1) = 0.69 , p < 0.05. This finding indicates that the use of video before face-to-face observations has a 


142 



positive influence on PSTs overall learning and use of new teaching concepts. These highly positive correlated 
significant results indicate that when PSTs review videos made by their peers in combination with their field 
observations, they are more likely to leant new teaching concepts and plan to apply them when becoming a teacher. 

The Spearman’s rho revealed a two-tailed statistically significant correlation between students who were 
able to make connections between the new concepts learned in the video prior to observing in a face-to-face 
environment and those who would recommend having students watch the video prior to entering the classroom, 
rs(l 1) = 0.67, p < 0.05. There was a strong, positive, and two-tailed statistically significant correlation between the 
learning about the different aspects of the first day of school from watching the videos and students preferring to 
have a blend between virtual video and face-to-face field experiences, rs(l 1) = 0.74, p < 0.01. These results show 
that when PSTs learn new teaching concepts through watching video related to observations on the first day of 
school, which helped them prepare for the actual face-to-face observation, are likely to prefer a combination of both 
virtual video and face-to-face field observations. 

One of the four open-ended questions asked students, “What was the most helpful part of the video field 
experience?” Students thought the videos provided information about how to create and establish control on the first 
day of school, “It was most helpful to see how everyone worked together in order for the day to run as smoothly as 
possible.” By having PSTs view the video, one stated that it helped eliminate the feeling of “being in the way.” This 
indicated that the video assisted in preparing the PST for the actual face-to-face observation and they felt more 
comfortable being in the classroom. Another PST remarked that the videos were aided in “Learning about the first 
day of school, since we were unable to see that!” The first day of school is a vital part of PSTs educational 
experience and the video provided students with a view that they were not otherwise able to see due to scheduling 
and timing of the year. 

PSTs were also asked about learning new concepts. Students listed classroom management techniques 
including, how to deal with a student who is upset when parents have left, and the importance of routines on the first 
day to set the stage for the rest of the year. One student stated, “I learned a little bit about how to get children that 
are having a hard time separating from parents [to] calm down and engage [them] in the classroom.” Others 
observed teaching methodologies and management techniques that can be introduced on the first day to encourage 
learning and structure such as, “correct circle procedure [and] how to raise hands.” One student stated: 

I have gained some strategies that I can use to help children make a smoother transition to starting their 
new school year. Such strategies were performed by caring, experienced teachers and effective 
routines/systems were established to ensure the best 'first day of school' possible. 

PSTs were then asked to explain how their teaching has changed. “Going over rules and not expecting to 
teach material on the first day as much as routine and rules so the rest of the school year goes smoothly.” Students 
were able to reflect on the first day of school to begin to understand that the first day of school sets the tone and 
structure for the entire year and to know that curriculum materials will probably not be given on the first day. 
Another student remarked, “It showed me how to establish control of a brand new classroom without being overly 
pushy or patronizing with the children. It made me realize that I had underestimated what children can understand.” 
Another response emphasized the importance of future application for when he encounters his first day experience: 

I just think it showed the organization on the first day of school. It is something to keep in 

mind for the future when I become a teacher and 1 can look back and remember how those teachers handled 

the first day and kept it organized. 

Additional important concepts about the first day of school were organization of the classroom and realizing that the 
teacher did not have to be patronizing by talking down to the students. PSTs realized the significance of being 
organized and allowing students the chance to learn the rules of the first day. These teaching and learning concepts 
related to the first day of school will assist pre-service teachers in their future classrooms. 

Conclusion 

An identified need within our teacher-training program was to provide field experiences around the first 
day of school to further develop our PSTs understanding and knowledge of this important learning environment. 
Research identifies this need as widespread and that it is important for PSTs to be trained to understand how to 
establish respectful and connected learning environments with and among students that begins on the first day of 
school. While we know that our students have been able to identify respectful and connected learning environments 
they were not able to experience how these relationships were established. Our findings identified that the PSTs 
gained knowledge about how to create and establish classroom control, behavioral management techniques, 
importance of routines, and new teaching methodologies. 
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As we continue to meet the needs of our pre-service teachers and re-examine our field experience 
opportunities it is clear there are needs that must be addressed and we hope to integrate technological tools to meet 
these deficiencies. We had an immediate need to fill the lack of exposure of our students to the first day of school 
and felt that our pre-service teachers were at a disadvantage if they took their field experience requirement during 
the spring semester. The results of our initial study were positive and provided statistical data to encourage us to 
expand the video field experience opportunities. 

What other needs might be addressed with a different approach to meeting the field experience 
requirement? As the research identified, there may be opportunities to increase access to quality classrooms, to 
encourage PSTs to explore new ideas in a safe environment, and to increase exposure to diverse populations and 
cultural backgrounds in schools beyond the current geographical locations. With the use of technology PSTs can 
experience observations of a greater set of master teachers, and a common shared ability to reflect on these settings 
(Hixon & So, 2009; Phillion et al., 2005; Karsenti & Collin, 2011; Fox et al., 201 1). As we continue exploring these 
opportunities we aim to add to the small body of research on the use of technology in field experiences and ways to 
meet the need to design experiences and curriculum around the first day of school learning environment. This is just 
one small piece of the growing research reflecting our changing definitions of education and teacher preparation. 
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Abstract 

This article presents empirical findings of a literature review focusing on the effects of gaming design characteristics 
on learning. Gaming characteristics (GCs) are believed to play a critical role in improving instructional quality of 
computer-assisted learning. In particular, researchers suggest that linking GCs with specific learning outcomes can 
harness the power of games and inform learning and teaching theories. While there has been a growing body of 
empirical research focusing on games’ learning effectiveness, the systematic knowledge linking GCs with human 
performance is extremely limited. Only nine empirical studies evaluating instructional benefits of singular GCs were 
found. Among the investigated GCs were storyline/fantasy, competition, reward, and interactivity with learning 
content. Research linking these GCs with specific learning outcomes suggests directions for future investigations. 

Background 

Empirical research has shown that well-designed games have a potential to improve academic achievement 
and engagement (Young, Slota, Cutter, Jalette, Mullin, Lai et al., 2012). Games have been used as a means to 
improve communication, present instructional materials, train, and motivate learners in a variety of settings, such as 
educational institutions, military, business and industry. 

Although there are many claims about instructional effectiveness of games, researchers do not have yet an 
answer on how to make fundamental decisions whether or not to consider games as part of an instructional solution. 
For example, learners’ audience, quality of a game, subject matter domain, instructional objectives, desired learning 
outcomes, or environmental constraints, are just a few factors that may influence educators’ decisions on whether or 
not to incorporate games in a curriculum. Popular video gaming websites (e.g., http://www.gamespot.com/ , 
http://www.gamespy.com/ ) offer a large variety of game categories to choose from including puzzles, action, 
adventure, shooters, role-playing, strategy, racing and driving, sports, platformers, fighting, simulations, etc. One of 
the major differences between the categories is the game design or more specifically gaming characteristics (GCs). 
The question is how to choose an appropriate game to achieve specific instructional objectives and what role GCs 
play in this process. 

There are many different perspectives on the mix of GCs that define a good instructional game. Many 
researchers agree that motivation is one of the major benefits associated with game-based learning (Barab, Gresalfi, 
& Ingram-Goble, 2010; Belanich, Orvis, & Sibley, 2006). Researches relying primarily on motivational benefits that 
games offer over other computer-based activities focused on identifying GCs that facilitate engagement and interest. 
Malone and Lepper (1987) and Malone (1981) suggested that challenge, control, curiosity, and fantasy are the major 
gaming characteristics that influence game player intrinsic motivation. More recently, Belanich, Sibley, and Orvis 
(2004) conducted an experimental study to determine motivational attributes of a successful PC -based game that has 
a potential to facilitate learning. The authors identified the following four motivational elements that contributed to 
participants’ interest to playing the games: challenge (not too hard and not too easy), realism (audio-visual realism 
and adhering to laws of nature), exploration (opportunity to discover new things), and control (manipulating the 
virtual environment through keyboard/mouse interface). 

Another set of thinkers considered all possible game features that may contribute toward a definition of a 
good instructional game. De Felix and Johnson (1993), for instance, viewed a game as a mix of dynamic visuals, 
interactivity, rules, and goals. Garris, Ahlers, and Driskell (2002) identified six characteristics of video games: 
fantasy, rules/goals, sensory stimuli, challenge, mystery, and control. According to Juul (2003), six elements 
compose a game: rules, variable quantifiable outcome, player effort, valorization of the outcome, attachment of the 
player to the outcome, and negotiable consequences. Vogel, Greenwood-Ericksen, Cannon-Bowers, and Bowers 
(2006) argued that the following five game attributes distinguish games from other types of computer-assisted 
instruction: motivation, reward, interactivity, score, and challenge. Gee (2009) proposed six elements that constitute 
a good instructional game: (a) an underlying rule system and game goal to which the player is emotionally attached; 
(b) micro-control that creates a sense of intimacy or a feeling of power; (c) experiences that offer good learning 
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opportunities; (d) a match between affordance (allowing for a certain action to occur) and effectivity (the ability of a 
player to carry out such an action), (e) modeling to make learning from experience more general and abstract, and (f) 
encouragement to players to enact their own unique trajectory through the game (p. 78). Whereas Shute and Ke 
(2012) identified seven elements of well-designed games: interactive problem solving, specific goals/rules, adaptive 
challenges, control, ongoing feedback, uncertainty evokes suspense and player engagement, and sensory stimuli (a 
combination of graphics, sounds, and/or storyline used to excite the senses) (p. #). This list of game "must haves" 
relied mainly on two descriptions of games: (a) as "a system in which players engage in an artificial conflict, defined 
by rules, that results in a quantifiable outcome” (Salen & Zimmerman, 2004), and (b) as a mix of game elements that 
includes rules, goals, interaction, outcomes, feedback, win states, conflict, competition, challenge, problem solving, 
and story (Prensky, 2001). Apparently, there is little consensus about all GCs that constitute a good instructional 
game. Yet, rules/goals, challenge, and control GCs are among the most cited GCs. 

Several researchers attempted to create a broad taxonomy of GCs and link GCs with specific learning 
outcomes in order to provide the basis for the decision of whether or not to consider games as part of an instructional 
solution and, if so, which GCs are appropriate from an instructional point of view. Johnson, Spector, Huang, and 
Author (2007), for example, suggested 14 GCs based on a comprehensive literature review: challenge, competition, 
rules, goals, fantasy, changed reality, immediate feedback, interaction, story or representation, engagement or 
curiosity, role playing, control, tasks, and multimodal presentation. The authors found that that 1) GCs are critical to 
successful game design in general, and that 2) the literature does link games (not GCs) to some learning situations 
(but not to specific instructional objectives or outcomes). Later, Wilson et al.’s (2009) literature review on game 
attributes resulted in identifying 18 GCs believed to have an impact on learning: adaptation, assessment, challenge, 
conflict, control, fantasy, interaction (equipment), interaction (interpersonal), interaction (social), 
language/communication, location, mystery, pieces or players, progress and surprise, representation, rules/goals, 
safety, and sensory stimuli. The authors analyzed games employed in the gathered studies in terms of GCs and 
learning outcomes that researchers aimed to achieve with the use of a particular game. However, vast majority of the 
reviewed games utilized a mix of multiple GCs, and as a result, it was impossible to determine the instructional 
benefits of singular GCs. Wilson et al. concluded that empirical studies are needed to examine which GC leads to 
which learning outcomes. 

The notion of effects of GCs on learning is equally important to design of digital learning environments and 
theories of learning. According to Lepper (1985), linking instructional objectives with GCs can help in refining 
learning and teaching theories. Similarly, Garris et al. (2002) stressed the importance of examining characteristics of 
instructional games. The authors state, “Although most agree that games can be engaging and that games can be 
instructive, there is little consensus regarding the essential characteristics of instructional games. Implicit in the 
research literature is the notion that if we pair instructional content with certain game features, we can harness the 
power of games to engage users and achieve desired instructional goals.” (p. 441). More recently, the Federation of 
American Scientists (2006) emphasized the importance of developing “a sound understanding of which features of 
games are important for learning and why, and how to best design educational games to deliver positive learning 
outcomes” (p. 5). 

By looking at the effects of different gaming characteristics and their interactions with student’s attributes, 
it is possible to identify GCs that make learning with computers appealing and motivating and provide practical 
recommendations for designing effective digital learning environments. Particularly, the effect of gaming 
characteristics on learning might depend on student’s traits or a desired learning outcome (Hays, 2005; Johnson, et 
al., 2007; Wilson, et al., 2009). 

Although many researchers have stressed the importance of examining the effects of GCs on human 
performance and linking GCs with learning objectives, empirical research investigating the relationships between 
GCs and specific learning outcomes is very limited. Probably, only feedback (Shute, 2008) and control over the pace 
of learning ( Azevedo, 2005) GCs were extensively examined empirically, as essential elements of well-designed 
computer-assisted learning environments. 


Gaming Characteristics 

The effect(s) of a particular GC on learning depends greatly on learning theories related to each GC. 
However, the way a GC is embedded into a game plays equally important role. The following sections present a 
review of learning theories and design issues related to competition and storyline/fantasy GCs. These two GCs were 
the focus of the majority of the identified empirical studies. 
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Competition 

Competition in digital games serves several purposes. It allows risk-taking in safe environment and contributes 
to motivation and skills testing, the enjoyment that was reported by many game players (Griffiths 1991a, 1991b, as 
cited in Lucas & Sherry, 2004). Competition has been cited among the major motivational gaming characteristics 
(e.g., Malone & Lepper, 1987; Prensky, 2001; Vockell, 2004). 

It is believed that students follow performance-oriented goals when an environment emphasizes social 
comparisons of ability and competition for grades (Meese, Anderman, & Anderman, 2006). Although some 
researchers argue that competition facilitates motivation and academic performance (Hromek & Roffey, 2009), 
adopting performance-orientated goals may also decrease motivation to learn (Meese, Anderman, & Anderman; 
Colquitt & Simmering, 1998). Particularly, low-achieving learners are the first to be handicapped in learning 
structures that encourage competition (Powell, Ames, & Maehr, 1990, as cited in Schunk & Zimmerman, 1994). 

Little consensus regarding the effect of competition on students' performance in classroom settings was found in 
empirical research as well. Colquitt and Simmering ( 1998), for instance, found that performance orientation was 
negatively related to motivation to learn. On the other hand, other studies showed the tendency to perform is 
stronger in the competitive than on non-competitive conditions (Horner, 1968; Rudow & Hautaluoma, 1975). 
Although the effect of competitive situations on individual performance has been extensively examined in classroom 
setting, extremely limited empirical research concerning the effect of competition on individual performance in 
instructional simulations or games has been found. Most of the literature examining digital simulations and games 
advocates a cooperative environment rather a competitive one, since competition may lower motivation of some 
students (Williams & Williams, 2007). 

Generally, there are four main types of competition that can be used to increase a learner's motivation in 
computer-assisted learning (presented in order of decreasing motivational effect for the average learner): (1) 
competition against other learners, (2) competition against the computer, (3) competition against oneself, and (4) 
competition against the clock (Alessi & Trollip, 2001). Although competition against other learners is the most 
motivating method, many educators avoid using competition among individual learners because it might affect highl- 
and low-achieving learners in different ways. Students who do well benefit from this competition type, while 
students who do poorly may experience feelings of embarrassment and punishment when competing against others. 

Mandryk, Atkins, and Inkpen (2006) explored users’ emotions in various collaborative play conditions. 
Twenty-four male participants played a sports game in three conditions: against a friend, against a stranger, and 
against a computer. They found that (1) it was considerably more fun and exciting to play against a fiend than 
against a stranger, and (2) it was considerably more fun to play against a stranger than against the computer. 
Marginal differences in excitement were revealed between stranger and computer conditions. The results of this 
study should be interpreted with caution, since a very small sample size of male participants only was employed. 

Storyline/F antasy 

Storyline was identified among the major reasons people play video games (Sherry & Lucas, 2003). Rieber, 
Davis, Matzko, and Grant (2001) define a storyline as a "glue" that connects scenes in instructional games. Although 
some researchers distinguish between storyline and fantasy GCs (e.g., Malone & Lepper, 1987; Prensky, 2001), for 
the purpose of analysis of the GCs, fantasy and a storyline are used mutually exclusively in this paper, since any 
fantasy is essentially a story (McKee, 1998). Storyline or fantasy has been identified (Malone, 1981; Malone & 
Lepper, 1987) among the prominent GCs that promote motivation to continue playing. The role of storyline or 
fantasy in gaming is to have players experience a kind of unreal activity. Malone (1980) distinguished between 
intrinsic and extrinsic fantasy as possible factors that may affect games effectiveness. According to Malone and 
Lepper (1987), an intrinsic fantasy fully integrates the fantasy context into instructional content. Effective storyline 
or intrinsic fantasy sustains players' engagement by stating clear objectives, providing a variety of paths to complete 
the objectives, and driving the player to act. 

Emotionally appealing storylines can motivate and engage, leading to increased learning (Habgood, Ainsworth, 
& Benford, 2005). Although, storyline is not powerful enough to sustain players’ motivation and engagement 
throughout the game (Myers, 1990, as cited in Asgari & Kaufman, 2004), it influences players’ decision whether or 
not to engage in the game play. The storyline hooks up players’ attention at the beginning stages of the game and 
lures the player into the game. In this way, the storyline facilitates an activation of other gaming characteristics such 
as interactivity, competition, control, curiosity, challenge, and feedback (Asgari & Kaufman, 2004). 

There is little consensus about the instructional effectiveness of integrating storyline into learning materials. 
Cordova and Lepper (1996) reviewed the literature on the effects of topical or content variables on children's 
curiosity and interest. They pointed out that children learn better, when materials are connected with topics and 
characters that are of high interest. 
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Habgord et al. (2005) analyzed theoretical and empirical foundations of intrinsic fantasy in the domain of 
instructional gaming and concluded that although fantasy does not play a critical role in improving the effectiveness 
of instructional games, its motivational power is a significant factor in enhancing motivation and learning in 
computer-based games. 


Purpose 

The purpose of the literature review was to identify empirical studies examining the effects of singular GCs 
on learning as opposed to a mix of multiple GCs on learning. Specifically, the goal was to answer to the following 
questions: (a) What GCs were empirically investigated? (b) How GCs affect learning? (c) Which GCs led to which 
learning outcomes? (d) What research framework researchers used to explore the effects of GCs on learning? 

Several GCs, i.e., feedback, control over the pace of learning, and assessment, were excluded from the literature 
review, since they were already extensively studied and reviewed in the literature (Ifenthaler, Eseryel, & Ge, 2012; 
Shute, 2008). 


Method 

A large body of literature concerning games and GCs was systematically reviewed. Online databases such 
as ScienceDirect, ERIC, Google Scholar, and Psychlnfo were searched for the relevant literature. However, most of 
the studies were found using manual search from references. The following search terms and keywords were used: 
games and learning, gaming characteristics, gaming attributes, game design, game design elements. 

The goal of the search was to find published and unpublished articles featuring empirical research on the 
effects of singular GCs on learning. In order to make the number of the reviewed articles manageable but 
comprehensive, exhaustive review with selected citation approach (Cooper, 1988) was implemented. Specifically, 
the following inclusion criteria was applied: 

1. The study explored how a specific GC affected learning. 

2. Feedback, control over the pace of learning, and assessment GCs were beyond the scope of the study. 

3. The study reported research methods (i.e., participants, instructional interventions, instruments, research design, 
etc.) and results. Studies that employed quantitative research method reported statistical analysis methods and 
results. 

4. The research presented in one study did not overlap research from another study. 

5. The article was written in English. 

The search was not limited to a particular publication type, date range, or research methods. From over 70 
initially collected articles, only nine studies were included in the literature review. 

Findings 


Rules/Goals GC 

Breakout: breaking bricks 

Malone (1981) attempted to find the “secret” for the engagement appeal of the Breakout game by 
examining three game features that seemed to capture the essence of the game: the score, the act of breaking out of 
the bricks, and the ball bouncing off the paddle. By systematically stripping down these three GCs and the 
combinations among them from the original game, six versions of the game were developed. The participants, 
college students, played the games and rated them based on their preferences. Malone found that the versions that 
excluded a score and “bricks breaking” features were significantly less appealing. While the game feature of “bricks 
breaking” refers rather to a combination of GCs than a singular GC, the author suggests that “bricks breaking” 
ultimately represent the goal of the game and the findings of his study suggests that without a clear goal, the game 
loses completely its gaming appeal. Although the instructional benefit of the Breakout game is somewhat vague, this 
study represents one of the earliest attempts to examine GCs and their engagement appeal. 

Storyline/F antasy GC 

Six empirical studies focusing on the instructional benefits of a storyline/fantasy GC or a combination of a 
storyline GC with another GC were found. In order to conduct the studies, the authors embedded an instructional 
content within a storyline to facilitate students’ engagement in the learning process. 
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Computer-based instructional programs for teaching basic graphics commands in Logo 

Parker and Lepper (1992) conducted two studies evaluating the effects of a fantasy GC on learning how to 
draw lines that connect various objects on screen with third- and fourth-grade students. In the first study, the authors 
created four computer-based instructional programs that that shared identical series of instructional tasks but 
differed in fantasy “embellishments” in order to examine the motivational benefits of different fantasy 
“embellishments”. In the basal, no-fantasy, version children drew abstract lines and geometric shapes. The other 
three instructional programs presented the same learning content in the context of simple stories such as a pirate 
seeking hidden treasures, a detective investigating a crime, and an astronaut looking for new planets. For example, 
in the no-fantasy version students were asked to connect a circle with other object on the screen. In the astronaut 
version, faces would replace the circles and a student would be guided to go and meet other astronauts. Simple 
illustrations were used to create the imaginary world. 

During the experiment, students (n=47) briefly reviewed the four programs, rated them, and were offered to 
play with the activity that they chose for 40 minutes. The results indicated that embedding fantasy elements into a 
learning content had a significant positive effect on children’ s motivation. The no-fantasy version had a significantly 
lower rating as compared to the other three fantasy versions. 

In the second study, Parker and Lepper (1992) examined the instructional benefits of the fantasy contexts 
explored in the first study. In particular, the authors focused on the immediate learning of materials, retention of 
materials, and understanding of general geometric concepts. A pre -/posttest experimental design was employed. 
Twenty-seven third-graders were assigned to one of the following three intervention conditions: (1) a no-fantasy 
control condition (similar to the first study) (n=7), (2) individualized-fantasy (n=10), and (3) assigned fantasy 
( n= 1 0). In individualized fantasy condition, children chose the fantasy context that they like the most. In the 
assigned fantasy condition, children were assigned specific fantasies. 

The results revealed that children in the fantasy conditions outperformed those in the no-fantasy condition 
on the immediate and delayed (2 weeks) learning of materials. Interestingly, on the general geometric concepts test, 
non-significant differences across the non-fantasy and fantasy conditions were observed. However, the students in 
the fantasy conditions performed significantly better on the delayed posttest 2 weeks later. In conclusion, Parker and 
Lepper (1992) state that including the least possible fantasy “embellishments” can result in better learning, retention, 
and motivation. 

Statistics simulation game 

Author (201 1) examined instructional benefits of a storyline GC on learning effectiveness, efficiency, and 
engagement with the use of an online instructional simulation for graduate students in an introductory statistics 
course. In addition, the study focused on examining the effects of a storyline GC on specific learning outcomes, i.e., 
factual, conceptual, and application knowledge. 

A storyline was embedded in an instructional simulation aiming to engage students in problem solving and data 
analysis in the context of basic statistics. The author developed two different versions of the simulation that had the 
same instructional content but differed in presence or absence of a storyline GC: (1) Simulation+No GC and (2) 
Simulation+Storyline GC. The Simulation+Storyline GC intervention presented learners with the Career Coach 
Simulation, where a student took on the role of an intern at the career advising company. As an intern, a game player 
had to assist a career coach with carrying out basic data analysis tasks. The storyline was presented using audio- 
based dialogs between the storyline characters and animations. 

A pre-/posttest experimental design was utilized. Graduate students (n=64) were randomly assigned to two 
intervention conditions. The results indicated that adding a storyline to a simulation did not result in significant 
improvements in learning effectiveness or efficiency. The analysis of students' performance scores for factual, 
conceptual and application knowledge did not reveal any significant differences between the interventions as well. 
However, students' performance from both interventions combined significantly improved from pre- to posttest. 
Both interventions (simulation and simulation with a storyline) showed significant learning gains related to 
application knowledge, thus supporting previous research showing a positive effect of simulations and games on 
developing higher cognitive skills. With regard to engagement, contrary to the authors' hypothesis, significantly 
higher engagement levels were observed among the students from the Simulation+No GC than the 
Simulation+Storyline GC group. 

The findings of this study suggest that adding a storyline may distract students from an instructional task and 
pose a higher level of extraneous cognitive load that may be detrimental to learning outcomes. Moreover, students’ 
preferences as well as the nature of the storyline and the storytelling system can affect students’ engagement in the 
learning process. 
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Cristal Island: a narrative-centered learning environment for teaching microbiology 

Spires et al. (2010) conducted an exploratory study aiming to determine the effects of different types of 
narratives on microbiology science learning and engagement levels with eight-grade students using a narrative- 
centered learning environment called Cristal Island. The virtual world of Cristal Island creates settings for a recently 
discovered volcanic island where a game player tries to solve a science mystery. The gaming environment allows a 
game player to explore the virtual world of the island, interact with game characters, operate lab equipment, 
manipulate objects, collect data, form and test hypotheses. All interactions in the Cristal Island virtual world are 
fully text-based and menu-based. Three versions of learning environments were developed to test instructional 
benefits of a storyline GC: Crystal Island Narrative (n = 60), Crystal Island Narrative-Light (n = 55), and Content 
Control (n = 36). Both the Narrative and Narrative-Light versions were developed using the Crystal Island game 
platform and enhanced with a storyline. The Narrative-Light condition used a storyline with less details included. 
The Content Control group used a PowerPoint presentation that featured content similar to the gaming environment 
curriculum, but did not include the storyline used in the gaming environment. 

Spires et al. (2010) used a pre-/posttest experimental design where 151 participants were randomly 
assigned to the three conditions. The results indicated that students from the Content Control group significantly 
outperformed students from the Narrative and Narrative-Light groups on the science content learning. Moreover, 
students from the Narrative-Light group reported a significantly higher engagement levels compared to the Content 
Control group, while no significant differences were found between the Narrative and the other treatment conditions. 
However, it is important to mention that a caution should be used when comparing Narrative or Narrative-Light 
versions with the Content Control condition. Since Narrative or Narrative-Light conditions featured interactive 
Cristal Island gaming environment, while the Content Control condition used a text-based PowerPoint presentation. 
Thus, it seems that the Content Control condition differed from the other two conditions in not only the presence or 
absence of a storyline GC but in interactivity and overall gaming appeal as well. In sum, the authors concluded that 
the Narrative condition posed excessive cognitive load on students, which negatively affected students’ learning. 

Storyline and Reward GCs 

Computer-based training program for teaching employment laws 

DeRouin-Jessen (2008) manipulated multi-media fantasy and reward GCs with the use of a computer-based 
training program featuring employment laws that govern selection practices content. The pre-/posttest experimental 
study systematically investigated the effects of multi-media fantasy (vs. text-based fantasy) and reward (vs. no 
reward) on learning and motivation. Five versions of computer-based training program were created: (1) Multi- 
media Fantasy, No Reward, (2) Multi-media Fantasy, Reward, (3) Text-based Fantasy, No Reward, (4) Text-based 
Fantasy, Reward, and (5) Traditional Version. The multi-media fantasy version was created using a commercial off- 
the-shelf (COTS) role-playing video game called Neverwinter Nights™, by Bioware. Neverwinter Nights™ game 
was adapted to allow players interact with the business owners and employees of various companies in a small town. 
The game characters communicated with the players using audio, while players responded by either typing or 
selecting responses to characters' questions from a menu. The text-based fantasy version was developed using 
Visual Basic and the communication between the characters and the players was in text-based format only. The 
traditional computer-based version was developed using Visual Basic as well. This version did not include any 
storyline, sounds, or graphics and the information was presented using a bulleted list format. The reward GC was 
facilitated by providing players with points in the form of a salary raise or promotion to a higher position. 

Declarative and application knowledge, motivation, and satisfaction levels were the focus of the study. One 
hundred sixty nine college students were randomly assigned to one of the five intervention groups. Contrary to the 
study hypothesis, traditional version, which looked similar to a PowerPoint presentation, led to better declarative 
knowledge outcomes than the multi-media versions. Moreover, adding a storyline or reward GCs to a training 
program led to non- significant differences across the conditions in motivation, satisfaction, and application 
knowledge. According to author, enhancing traditional learning materials with GCs can distract learners and result 
in a lower performance. In sum, adding a media-based storyline resulted in a lower declarative knowledge, while 
adding a reward GC had no effect on learning or motivation. 

However, it is unclear whether the only difference between the multi-media fantasy and text-based fantasy 
versions was the storyline presentation format (media-based vs. text-based) or game navigation, interaction, and/or 
appeal contributed to the differences between the versions as well. Since these two instructional interventions were 
created using different game platforms (Neverwinter Nights™ vs. Visual Basic). 
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Storyline and Interactivity with Learning Content GCs 

Synthetic learning environment for teaching African American history 

Greenwood-Ericksen (2008) examined the effects of a storyline (present vs. absent ) and the degree of 
interactivity with the learning content (high vs. low) on learning. The author created a synthetic learning 
environment that included a hybrid of games and simulations for teaching African American history topics on 
slavery and the Underground Railroad to college students. The storyline GC was manipulated by presenting a story 
either through a narrative (storyline present) or text-based list of facts (storyline absent). The degree of interactivity 
with the learning content was manipulated by presenting the events either through the synthetic learning 
environment (high interactivity) or through text only (low interactivity). The “Story with Interactivity” condition 
was created using a COTS computer game called Neverwinter Nights ®, by Bioware, that allowed students to 
interact with the characters and drive the plot development. The author did not provide technical details on how the 
other treatment conditions were developed except by stating that students interacted with a “limited” synthetic 
learning environment (p. 36). 

Content learning (measured through recall and recognition questions) and enjoyment were the focus of the 
study. Eighty participants were randomly assigned to the four conditions. After interacting with the synthetic 
learning environment, the students were evaluated on the content learning. The results revealed that students from 
the “Storyline with Interactivity” condition performed more poorly than the other intervention groups; but their 
enjoyment was the highest among the four conditions. The presence or absence of a storyline did not affect students’ 
performance when interactivity was low. However, when interactivity was high, a combination of a storyline with 
high interactivity negatively affected students' learning. In sum, the degree of interactivity with the learning content 
did not have any effect on learning. While, adding a storyline GC negatively affected students’ performance. With 
regard to enjoyment, both GCs, i.e., a storyline and interactivity with the learning content, had positive effect on 
students’ enjoyment. 

Storyline, Feedback, Sensory Stimuli, and Competition GCs 

Darts: a cognitive skill game for teaching mathematics 

Malone (1981) used Darts, a game for teaching elementary students fractions, to examine students 
preferences with regard to various GCs. In the game, three balloons are shown on a number line and students need to 
guess the balloons position by typing fractions. If a student guesses the balloon position correctly, an arrow pops the 
balloon on the number line. Seven GCs were stripped down from the original version of Darts to examine their 
effects on students’ interest to play the game: performance feedback, scoring, constructive feedback, extrinsic 
fantasy, music, graphic representation, and intrinsic fantasy. The author distinguished between extrinsic fantasy that 
weakly related to the skill being taught (an arrow pops a balloon not on the number line but in another part of the 
screen) and intrinsic fantasy that “intimately related” to the skill being taught (an arrow pops a balloon on the 
number line). 

A very small sample size of 80 students was employed in the study. The participants were randomly 
assigned to the eight conditions. The results revealed two significant findings. First, adding the extrinsic fantasy of 
balloons and arrows significantly improved students “liking Darts.” Second, a significant interaction between sex 
and condition was found. Adding the intrinsic fantasy to the game resulted in significantly lower interest toward the 
game among girls. Malone (1981) suggests that fantasy plays an important role in creating motivating learning 
environments. However, if the fantasy does not appeal to the target audience it may decrease students’ interest in 
learning. 

Competition and Contextualized Advisement GCs 

Simulation game for teaching mathematics 

Van Eck and Dempsey (2002) systematically examined how competition and contextualized advisement 
GCs affect transfer of mathematics skills. An exploratory computer-based simulation game was developed for this 
study using Macromedia Authorware. In this simulation game, a learner plays a peer-aged character helping her 
“aunt and uncle” to fix up a house. Learners need to calculate, for example, how much paint is needed for painting a 
room in a house by operating various mathematic concepts such area, perimeter, addition, subtraction multiplication, 
division, and number conversion. Contextualized advisement involved aunt and uncle characters providing video- 
based advisement about solving the problem encountered during the game. By clicking on a walkie-talkie icon 
students could “call their aunt and uncle from another part of the house” and get an advice, thus activating the 
contextualized advisement video. Given these features of the contextualized advisement, it is possible to conclude 
that contextualized advisement is a type of elaborated on-demand feedback (Shute, 2008). In the competitive version 
of the game, players compete against a computer character. 
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One hundred twenty seven middle school students participated in the pre-/posttest experimental study. 

They were randomly assigned to one of the five intervention conditions generated by crossing the two independent 
variables, i.e., competition and contextualized advisement, and adding a control group. The control group received a 
computer-based tutorial. The questions in the tutorial were text-based and identical to those in the other four 
conditions but did not have any graphics. 

An interaction between the competition and contextualized advisement was observed. Students in the 
noncompetitive group performed better when they had an access to contextualized advisement, while students in the 
competitive group performed better when they did not have an access to contextualized advisement. Van Eck and 
Dempsey (2002) suggest to interpret the results with caution due to several study limitations such as low statistical 
power, assessment issues, and technical glitches. In general, the results may imply that attending to elaborated 
feedback while trying to beat the game does not complement each other. Elaborated on-demand feedback, i.e. 
contextualized advisement, may increase cognitive load, thus lowering students’ ability to follow the competition. 
Conversely, students competing for a higher score do not want to lose their time on attending to and processing of 
contextualized advisement. 

Virtual Reality GC 

VR-ENGAGE - a virtual reality game for teaching geography 

Virvou et al.(2005) explored the benefits of virtual reality GC for learning geography with a virtual reality 
educational game called VR-ENGAGE. VR-ENGAGE is an Intelligent Tutoring System (ITS ) that consists, 
similarly to any ITS, of lour components: the domain knowledge, the student modeling component, the tutoring 
component, and the user interface. In this study, the authors manipulated the user interface component. A game-ITS 
version operated through a virtual reality user interface, while ITS with a simple user interface did not have the 
virtual reality features. Similar to the popular game called “DOOM” (ID-Software 1993), VR-ENGAGE has many 
virtual worlds with mediaeval castles, underwater castles, passages through the file, secret temples, dungeons and 
dragons where the student has to navigate through. VR-ENGAGE communicates with students through voice-based 
animated game characters or windows with text-based messages. Users can communicate with the gaming 
environment by typing their answers in a dialogue box. The ITS with a simple user interface did not have any 
features of the virtual reality. The system communicated all information through forms, dialogue boxes, hypertext, 
drop-down menus, etc. 

The experiment consisted of two parts that were carried out simultaneously. During the first part of the 
experiment, 90 fourth-grade students were randomly assigned to either game-ITS or ITS with simple user interface 
condition. During the second part of the experiment, 90 fourth-graders were divided into three groups - poor 0i=30), 
average (n=30), and good (n=30) - based on the grade they received in geography in the previous term. Students in 
each of the three groups were randomly assigned to either game-ITS or ITS with simple user interface condition. A 
pre -/posttest research design was employed. The results of the first part of the experiment indicated that VR- 
ENGAGE ITS led to significantly higher learning gains in comparison to the ITS with simple user interface. The 
results of the second part of the experiment revealed that students from the poor performance group that learned 
with VR-ENGAGE benefited the most of the all conditions. Moreover, students from the average performance 
group that used VR-ENGAGE outperformed students from the respective performance level group that used ITS 
with simple user interface. Finally, non-significant differences in students’ performance were observed between the 
good students from the VR-ENGAGE and non-game ITS conditions. 

In conclusion, a positive effect of virtual reality GC was observed when compared to the ITS with simple 
user interface. In addition, when students’ prior knowledge was considered, virtual reality GC benefited poor 
performers the most, whereas it had no significant effect on good performers. According to the authors, these 
findings suggest that virtual reality GC can provoke an interest in the subject of study among the students who tend 
to perform poorly in class and thus increase their learning, whereas good students perform well under any 
circumstances. However, no motivational data were collected to further support this claim. 

Summary and Discussion 

The literature review revealed an extremely limited number of empirical studies examining the effects of 
GCs on learning. Only nine studies were found. Several studies linked GCs with specific learning outcomes such as 
declarative, factual, conceptual, application types of knowledge (DeRouin-Jessen, 2008; Author, 2011), and 
knowledge transfer (Van Eck & Dempsey, 2002) or general learning benefits such as motivation, engagement, and 
retention (Malone, 1981; Parker & Lepper, 1992). The paucity of empirical investigations systematically linking 
GCs with learning outcomes suggests that more empirical studies should be conducted in this area. It is important to 
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mention that the goal of investigating singular GCs effects on learning is not to “dissect” successful games. Rather it 
is an attempt to identify gaming design elements that help achieve desired learning outcomes and/or make learning 
with computers appealing and motivating, and consequently provide practical recommendations for designing 
effective digital learning environments. 

The reviewed studies linked GCs with various learning outcomes across a variety of domains (see Table 1). 
Interestingly, six out of the nine gathered studies examined a storyline/fantasy GC. A storyline/fantasy GC was 
embedded in learning environments designed to teach mathematics, statistics, microbiology, graphic design, 
employment laws, and history. The analysis of finding demonstrated contradictive effects of a storyline/fantasy GC 
on learning. Some studies showed a positive effect, while others demonstrated negative or no effect on learning. 
Nevertheless, the researchers agree that a storyline/fantasy GC has a strong motivational effect and non-appealing 
storylines/fantasies can detrimentally affect students’ motivation. A closer examination of various types of storyline 
and fantasy elements used in the gathered studies revealed that storylines represent a quite broad concept. Stories 
can serve a major role in the learning process by guiding the presentation of the learning materials via audio-based, 
text-based, or interactive multi-media based narratives (e.g., DeRouin-Jessen, 2008; Greenwood-Ericksen, 2008; 
Author, 201 1) or just provide an idea about the location or settings where the instructional activity takes place 
(Malone, 1981; Parker & Lepper, 1992). Moreover, researchers were also concerned with the amount of details a 
story should provide and how it may affect learning (DeRouin-Jessen, 2008; Spires, et ah, 2010). In addition, there 
are many ways in which storylines can be presented, including multi-media (graphics, music, and interactions with 
the gaming environment) or text-based only. Apparently, a deeper investigation on the effects of storylines in digital 
learning environments should be attempted. 

A study of Virvou et al. (2005) illustrates further the broad range of storyline representations. The authors 
examined the effects of virtual reality created by a combination of story, audio, and visual elements on learning. 
According to the authors, adding virtual reality component to an intelligent tutoring system was beneficial for 
learning geography concepts, particularly for low-performing students. 

Interesting findings related to the effect(s) of GCs combinations were found. For instance, combining a 
storyline with highly interactive media was found to significantly decrease learning gains (Greenwood-Ericksen, 
2008). While interacting with the learning content without a storyline contributed to higher players’ enjoyment. 

Competition GC (sometimes represented as a score) was investigated in several studies (Malone, 1981; Van 
Eck & Dempsey, 2002). According to the results of these studies, competitive learning environments do not affect 
learning. However, combining competition GC with contextualized advisement GC appears to result in an ill- 
designed instructional game, since students in competitive learning environments are more concerned with getting a 
higher score or a better time than receiving an elaborated feedback regarding the instructional task (Van Eck & 
Dempsey, 2002). 

There are many challenges in exploring the effects of GCs on learning. In order to examine the 
relationships between the GC and learning, it is important to understand how each particular GC affects learning 
and/or specific learning outcomes. This means that GCs need to be studied in isolation from each other. This is a 
very difficult task, since commercial off-the-shelf games usually incorporate multiple GCs and do not allow code 
altering. In addition, research design and sample size considerations affect the number of GCs that can be 
systematically investigated in a particular study. These facts can probably explain the paucity of empirical research 
linking the GCs with human performance and the limited number of GCs explored in each particular study. 

The literature review findings underscores the challenges associated with carrying out this type of research. 
The majority of found studies used relatively small sample sizes thus affecting the generalizability of the findings. 
Moreover, all reviewed studies employed unique learning environments that were specifically created to address 
research questions in each study. Researchers either systematically manipulated one or two GCs, or stripped GCs down 
from an original game. More information about how to manipulate particular GCs can be found in Malone (1981). 
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Table 1 

Effects of GCs on Learning Outcomes* 

GC(s) Learning Outcome/ Benefit Effect 

DeRouin-Jessen (2008) - Employment Laws 



- Declarative Knowledge 

4 

Fantasy 

- Application Knowledge 

= 


- Motivation 

= 


- Declarative Knowledge 

= 

Reward 

- Application Knowledge 

= 


- Motivation 

= 


- Declarative Knowledge 

= 

Fantasy & Reward 

- Application Knowledge 

= 


- Motivation 

= 

GC(s) 

Learning Outcome/ Benefit Effect 

Greenwood-Ericksen (2008) — African American History 



- Overall Performance 

4 

Storyline 

(recall and recognition) 
- Enjoyment 

t 

Interactivity with Learning 
Content 

- Overall Performance 

— 

(recall and recognition) 
- Enjoyment 

t 

Storyline & Interactivity with 
Learning Content 

- Overall Performance 
(recall and recognition) 

- Enjoyment 

4 

t 


Malone (1981) - Mathematics 


Fantasy 

- Interest toward the game 

T 

Feedback 

- Interest toward the game 

= 

Score 

- Interest toward the game 

= 

Sensory stimuli 

- Interest toward the game 

= 

Malone (1981) - Breaking Bricks 


Goal 

- Interest toward the game 

t 

Novak (2011) - Statistics 


- Learning Effectiveness 

= 


- Learning Efficiency 

= 

Storyline 

- Factual Knowledge 

= 

- Conceptual Knowledge 

= 


- Application Knowledge 

= 


- Engagement 

4 

Parker and Lepper (1992) - Graphic Design 



- Overall Performance 

T 

Fantasy 

- Retention 

T 


- Motivation 

t 
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Table 1 (continued) 



Storyline 

Spires et al. (2010) - Microbiology 

- Overall Performance 

4 

t 

- Motivation 


Van Eck and Dempsey (2002) - Mathematics 

Competition - Knowledge Transfer = 

Contextualized Advisement - Knowledge Transfer = 

Competition & Contextualized - Knowledge Transfer attending to contextualized 

Advisement advisement while trying to 

beat the game does not 

complement each other 


Virvou et al. (2005) - Geography 

Virtual Reality - Overall Performance | 

Note. * Significant effects only. j - positive effect; J, - negative effect; = - no effect 
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Abstract 

Learning initiated by complex simulated tasks involves explorative thinking and inductive as well as 
analogical reasoning. Simulations of complex processes and systems require complex problem solving. 

Tasks need to be specifically designed to provide a struggle that holds on for long enough, so that 
over-generalizations will be abandoned by the learners. To meet that demand, we created a software 
prototype that simulates individual learners for students who enter the complex field of individual 
learning coaching. As a new development of the software we introduce personal stories from the past 
by the simulated (virtual) learner. In this paper we show how such stories can be integrated into an 
existing text-matrix-based approach. Two evaluation studies showed that the stories particularly helped 
with understanding the tasks and the underlying theory. 

Keywords: automated task synthesis; simulation; story-telling; scaffolding; accommodation 

resistance 


Background 

Simulation has become a widely used means for training learners, especially when they have to deal with complex 
environments and decision on their jobs (Thurman, 1993). Jones (1980) defines a simulation as “a working representation of 
reality.” Further, a simulation may be an abstracted, simplified, or accelerated model of a process or system which allows 
exploration where reality is too expensive, complex, dangerous, fast, or slow (see de Freitas, 2006; Garris, Ahlers, & 
Driskell, 2002). Within simulated environments a variety of cognitive processes regarding is to be expected just as for a real 
environment (e.g. Gentner & Stevens, 1983; Johnson-Laird, 1983; Rumelhart, 1980; Schank & Abelson, 1977; Seel, 1991). 

Learning initiated by complex simulated tasks involves explorative thinking and inductive as well as analogical 
reasoning. Simulations of complex processes and systems require complex problem solving. Complex problem solving 
requires iterative steps of hypothesis testing as well as increased time for constructing appropriate mental models (Funke, 
1992; Norman, 1983; Seel, Ifenthaler, & Pirnay-Dummer, 2009). Mental models are constructed in order to hypothesize and 
understand the structure of the simulation process or system and to simulate transformations of these processes and systems 
mentally (see Seel, 2003). The same processes that work in a real environment can be utilized to create learning 
experiences that are similar to the performance in the real world as regards complexity, texture and fuzziness (Pirnay- 
Dummer, Ifenthaler, & Seel, 2012). 

Methods of cognitive conflict seem to be the most promising in order to train learners for later on complex decision 
making and problem solving so far (Schnotz & Preuss, 1997; Seel, 1991, 2003). Tasks need to be specifically designed to 
provide a struggle that holds on for long enough, so that over-generalizations will be abandoned by the learners 
(Zangemeister, 1976). 
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There are many approaches to world-like complexity. Sometimes the resources or instructional means require an 
approach that can be integrated into classical settings of learning. Learning tasks provide a variety of opportunities to think 
things through differently, especially if they require a change of reasoning or thought. In learning environments of all kinds 
it has always been a challenge to provide learners with the right cases and tasks to train their abilities for transfer (Biggs, 
1999; Seel, 1992). There are collections of tasks available to teachers for numerous topics. Some of them are of course more 
oriented towards assessment (Resnick & Resnick, 1996). But there are also collections which aim at tasks that support 
learning (Danielson & Marquez, 1998; Willis, 2005). Complex tasks with a real surface are often needed to train a real 
situation (Savery & Duffy, 1996). If the structure of such a task becomes complex, it may create illusory contradictions due 
to the many variables that influence the model. Resolving such contradictions and using them to construct new knowledge is 
considered an important factor for learning (Schnotz & Preuss, 1997; Seel, 1991; Seel, 2003). Moreover, if the variables are 
not presented to the learner directly but rather create or control an environment, then they might be represented in a way that 
is fuzzy to the learner. This can be used to train solutions, classifications, and interpretations of feedback that would also be 
fuzzy in the real world. 


Learner-Model-Simulation (L-MoSim) 


To meet that demand, we created the L-MoSim software prototype that simulates individual learners for students 
who enter the complex field of individual learning coaching (e.g., helping individually with suitable learning strategies, 
motivational trainings, and even curricular planning to achive a certain qualification goal). L-MoSim simply stands for 
“Learner Model Simulation”. Within L-MoSim a virtual learner is simulated on a model of 24 scales that are both central to 
a learner and important for coaching as well. Each of the scales is represented by a number of different items, the same way 
as on a good questionnaire or a standardized test. The categories (scales) within L-MoSim are: interest, professional interest, 
motivation, diligence (industriousness), preciseness, ambition, preferred learning strategies, cognitive style, intelligence, 
general performance, exam nerves, anxiety, extraversion, introversion, self-efficacy, self-concept, empathy, emotionality, 
ability to communicate, behavior towards authority and seniors, conflict behavior, creativity, attitude towards schools and 
education, and openness. At the end. the software processes a case with a task. The case is represented as an anamnesis 
(case history). Additionally L-MoSim provides simulated quotes from an “interview” to illustrate interdependencies within 
different traits (e.g. motivation, anxiety, introversion). The task consists of the virtual leaner asking for help with his or her 
current situation and a specification aiming at a specific weakness of the virtual learner. 

Table 1. Simple one -dimensional text matrix (Motivation). 


Motivation 1 


Statement The learner has a 

The learner has a 

The learner 

has 

a 

The 

learner has a 

The learner has a very 

very low 

low motivation 

moderate 



high 

motivation 

high motivation and 

motivation and is 

and can only 

motivation and 

is 

and 

is always able 

participants in many 

almost unable to 

rarely concentrate 

able focus 


on 

to 

focus on 

additional learning 

focus on his tasks 

on his tasks. 

assignments 

most 

assignments. 

activities besides the 

at all. 


of the times. 





assignments. 


Table 1 shows an example for a single item of a trait relevant for learning within L-MoSim. There are more items 
like this to resemble a single category like motivation. 104 of such items are selected alongside the data-driven model and 
composed into a coherent text that creates the complex individual task for the students. More scales of content then form the 
12 current text-matrix tables of L-MoSim that are used to texturize the numerical simulation model. 

The users get an 18-page prose-text per case and a task description from L-MoSim, which is generated as a PDF- 
file by the software. The students will then have to come up with solutions. L-MoSim cases involve alleged contradictions 
that can only be resolved through integration of the different traits and intensified studies of the respective literature. Hence, 
in order to resolve L-MoSim cases, students have to demonstrate their ability to grasp complex problems. However, the 
alleged contradictions sometimes feel like real logical contradictions, especially when the prior theoretical knowledge is 
rather low ( Pirnay-Dummer, et al., 2012; Seel, et al., 2009). If such contradictions would happen in real cases (not the 
simulated ones), it would of course be almost impossible to ignore them. With a computer-generated case at hand, unsolved 
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parts of the problems can be more easily projected to software that may be flawed. In the learning process, what is new 
often contradicts what is already known. There is a natural resistance towards changing ones beliefs for something new 
(Seel, 1991). We call this effect accommodation-resistance, the resistance to accommodate new aspects within the mental 
structure and by this giving up knowledge and beliefs that are already there. And we actually saw this effect in prior studies 
(see Pirnay-Dummer, 2010). In order to help the students solving this kind of puzzle, we use two strategies: 

1 . Increase the feedback during the problem solving phase. By 

a. giving details and examples how the software works 

b. helping during the problem solving with specific questions concerning the case. 

2. Increase the plausibility of the case by introducing more texture like 

a. original quotes by the simulated learner 

b. additional stories about the simulated learner 

c. comments by the simulated interviewer who subsumes the results into the statements that are given by the 
simulation. 

In its current state, L-MoSim generates 1,08 10 16 different cases - all of them resembling the parameters and 
distributions of current research data and theoretical understanding of the involved internal and external variables of the 
learners. It spans 105 personality categories, 55 goal x vita configurations, five social backgrounds, 333 educational goals, 
two additional task specifications per category, 371 learner-quotes, 450 different stories, 35 inter-categorial difference 
interpretations, and 97 different fields of interest for the simulated learner to select from. Thus, real research shapes the 
simulation model that leads to the cases and tasks. L-MoSim first creates a numerical model that is aligned with multiple 
known correlatives between variables by working its way through a spreading activation model of all the involved variables. 
Hence, only the first numerical value is independent and completely random. All following variables depend on the 
relations between the variables and their distributions. But the numbers are not printed in the resulting document. The 
document contains text that is triggered by the numerical model. This does not only add the necessary fuzziness as 
described above but it also creates a type of texture that is closer to how cases are experienced in the real world: by verbal 
descriptions that are in many ways structured like concluding hypotheses from individual observations. 

Earlier versions of the software have already been evaluated by students with promising results. In our mixed- 
method evaulations and our post-hoc analyses, we found that learners wanted to have more texture to help them to think 
about their specific case. As a new development of the software we introduced stories from the past by the simulated 
(virtual) learner. The software now provides model-congruent stories to help the learner understand how different traits of 
the learner came into place. Also, beyond the original verbal descriptions of the simulated learners’ traits (see Table 1), the 
participants now get stories that fit the simulation model. 

Telling Stories within L-MoSim 

A new innovation to L-MoSim is the integration of stories that tell something about the simulated learners past. 
They are supposed to help the learners to understand how the traits of the simulated learner may have developed or how 
they effected the learners experiences in a time where he or she still went to school. 

Up to now, stories were implemented for several of the 24 existing scales within L-MoSim and deduced from 
theory such that students working on the case would find textbooks to back up their thoughts on the case: 

Extraversion , conscientiousness and openness belong to the Big Five constructs of personality (e.g., Asendorpf, 
2007; Borkenau & Ostendorf, 1993; Costa & McCrae, 1992; Digman, 1990; Eysenck, 1991). Introversion is in the Big Five 
model (Five Factor Model, FFM) the opposite pole to extraversion, but it is important, that introversion is not essentially the 
contrary of extraversion. Introversion is not interpreted as the lack of extraversion ( Saum-Aldehoff, 2007, p. 55). Thus 
introversion is implemented with a separate scale within L-MoSim. 

Creativity, communication, conflict behavior and the behavior towards authority are the interactive scales within L- 
MoSom. The stories have been created according to basic theories of the field (e.g., Bischof-Kohler, 2011; Brown, 1989; 
Feger, 1978; Hayes, 1989; Strohner, 2006). 
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No doubt that interest, motivation, self-efficacy and self-image play important roles in the processes of learning. 
Thus we also selected this field from the scales of L-MoSim and constructed stories for the different scales. Again, these 
stories have been created deductively from fundamental theories (e.g., Bandura, 1997; Deci, Koestner, & Ryan, 1999; Deci 
& Ryan, 2000; Heckhausen & Heckhausen, 2008; Kuhl & Heckhausen, 1996; Rheinberg, 2008; Schiefele, 1987; Usher & 
Pajares, 2009). 

The stories have been created in a way that they are as independent as possible throughout the scales of L-MoSim. 
Some stories however exclude each other and they do so also in the database of L-MoSim. The stories are selected if a scale 
has sufficient homogeneity. A maximum of four stories is written into the cases categories and exclusions permitting (mean 
number of stories = 3.3). 

Examples from the Stories 

We show two examples of raw stories and how they are implemented in L-MoSim. The software substitutes FN 
with the given name and SN with the surname later on. All gender-specific content is selected by the sex of the simulated 
learner. The placeholder AGE is substituted by the age the learner supposedly had when the story happened. Within the 
cases, all the learners are currently adults. The age is selected out of a possible range that is specific to each story. 

The first example is a story on low diligence (possible age ranging from 12 to 14). 

During an anamnesis session FN SN told the following scene, which happened at biology lessons, back when 
{she/he) was AGE years old: 

The client once remembered {her/his} biology teacher who assigned a task in class, to compile a herbarium. To do 
so, the class should collect und squeeze ten meadow plants. Every plants' features should be noted on a sheet of 
paper and for every plant they defined by family and kind. The information should contain the place, where the 
plant had been found as well as the name of the pupil and the data, when they found the respective plant. They had 
five weeks to compile the collection of plants, FN neglected the task almost completely. {She/He} said, that 
{she/he) had no time schedule whatsoever, because the timeframe was very tight to begin with and she didn’t feel 
to have enough time to complete the whole task. FN postponed the intention of collecting plants very often and had 
no personal desire to complete the herbarium. Because FN was not particularly interested in {her/his} task, 
{she/he} submitted on the settled day a collection of plants with only five squeezed meadow plants. 

The second example is a story on high creativity (possible age ranging from 9 to 14). 

During an anamnesis session FN SN told the following scene, which happened at biology lessons, back when 
{she/he} was AGE years old: 

On a Monday morning FN SN went exercising where also the most recent competition series were analyzed. 
During the last four days FN has been to five cities in order to compete against other sportsmen throughout the 
nation. Since the team did well, the athletes were allowed to create their own exercise program for their exercise 
unit. Together with two team members, FN drafted an obstacle course. This course included a slalom course that 
required high skill as well as different sections, which could only be managed with sufficient speed. Moreover, FN 
SN created even two more courses that were popular with the other team members, but they also were physically 
challenging. The exercise unit was fun for every sportsman involved. 

The introduction is always generic to shift the view from the simulated learner to the coach who is retelling the 
story. The stories are all about that size to be concise enough but still fit into the whole document structure. This method 
makes the occurrence of the story more plausible because it provided a background why the story came up in the first place, 
i.e. as an example for something of importance to the learners’ current situation. Not just some random story of the past. 
Thus, another “person” already picked that story Also, except for the quotes of the simulated learner, all other text is written 
as a reflection by the person who did the anamnesis for many different reasons, among them; to provide a rational why the 
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text even exists, to make it easier to “hide” reflections, to discuss the difference between hypothetical interpretation and real 
personality traits with the learners. 

Selecting the Right Stories Within L-MoSim (Algorithm) 

Within L-MoSim, the stories are picked and distributed to the case-text on the basis of all the available categories 
K g . Stories can only be applied to categories that have sufficient homogeneity, i.e. low variance within the category. For 
high-variance categories L-MoSim provides a different type of text output: Simulated “original” quotes by the learner. The 
quotes address the differences within a category and make them more plausible to the readers. A story for a category only 
makes sense if its items do not vary that much: 


K mv = VKg -> SD(K) < 0.6 

Thus, from the possible categories only the K mv categories are selected that have an internal standard deviation of 
less or equal 0.6. K mv are categories that can be selected for that particular numerical case of the model. If possible, four 
categories are selected from the case. Thus, up to four stories are picked for each case. The mean of the category leads to the 
specification of the story. It can be (a„= 1) on the lower end, average (a„=2) and high (a„=3). L-MoSim works on 5-point 
scales from -2 to 2, thus the specification a g of the story would be: 

fl ,M{K mv ,d < -1,6667 
^ = ] 3, M (K mV i) > 1,6667 
V. 2, else 

Individual stories may exclude each other by content. For instance, if one story tells that the learner enjoyed math 
during school, another story cannot say otherwise. The exclusions are not about something like high versus low motivation, 
because the L-MoSim software already takes care of model-consistency. It is rather about contradictions that may occur 
with the context and general texture of the story that are not part of the simulation model. A relational database on the 
exclusions was created during authoring. Because the story-combinations are very complex, specific authoring-tools were 
developed to help the authors find and reflect on the exclusions with the goal of sorting them out if possible. The authors 
were provided with current graphical information about the exclusion structure and with tables in which they can look up 
their stories. 

The overview in figure X shows a whole representation of a the existing exclusions at developing stage of the 
stories. This information was updates frequently and automatically by the L-MoSim developing system. The authors can use 
the graphical information to identify stories that do not go well with many other stories by looking at the centrality (number 
of incoming links) of the conclusions. They can then use tables to navigate through the exclusions and also see whether the 
exclusions are directional (isomorphism) or not (homomorphism). The authoring tools helped to speed up the overall editing 
and helped a lot with compatibility. A platform like this can be easily adapted and could be interesting for projects that deal 
with lots of content developing by different authors. 

Overall, the content of L-MoSim is specific to the field of learning and instruction but it’s simulation principles are 
generic and can be easily applied to other domains once a research- and data-based model is constructed for the specific 
domain. 


Research Interest (Evaluation) 

L-MoSim as used in this paper is a revised version of the program that was already in use and evaluated in several 
classes in 2009 and 2010. Within the first prototype version the students had troubles coping with the alleged contradictions 
of the cases (among other problem that are already solved in the current version). Despite the repeated discussions (over the 
course of the seminar) on how those contradictions are an inherent feature of L-MoSim and resemble real-world 
distributions, many of the students supposed that they could not be resolved. Instead the findings of the respective 
evaluation revealed that the existence of said contradictions had been attributed to the software. 
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To the learners it was easier to attribute the accommodation resistance to the software that “somehow fails to 
simulate the real world’s behaviour” than to draw the same conclusion on a real case. In retrospect, this simulation-specific 
problem seems only logical but we did not think of this aspect when we first developed L-MoSim. The qualitative and 
quantitative findings of the evaluation of L-MoSim in 2009 showed an accommodation resistance like that among many of 
the participating students. In our current study, we tried to use specific inductive and deductive scaffolds to provide the 
learner with a means to overcome this barrier within his learning process (Azevedo, Cromley, Winters, Moos, & Greene, 
2005). Specific scaffolding interventions are known to encourage the creation of mental models by helping the learner to 
build his own problem-solving scaffold (Seel, 2003, p. 373). 

Based on the assumptions of the effect of stories within automated task synthesis our research question towards the 
evaluation of the software is: Will the additional stories as part of the learner-simulation be considered an additional help by 
the students who use the tasks and help them with their studies? 

Methods 

The methods on the two studies are almost the same. The pilot study was conducted with 7 (5 female and 2 male) 
master students from social work / social management in their second semester. Their mean age was 23.9 years (SD = 2.87). 
The study took place during a seminar on individual learning coaching in 2011 in which L-MoSim cases and their tasks are 
part of the graded course assignments. The students worked on two cases during the semester and their work is part of the 
course grades. The L-MoSim cases were introduced after four theory lessons. The students were given a case from L- 
MoSim that contains the stories and the same case without the stories to directly compare them. All other features of the 
cases were the same. 

Over the course of the seminar the participants completed a modified version of the Heidelberg Inventory for 
Course Evaluation (HILVE) (Rindermann & Amelang, 1994) on two cases in order to determine the differences to the new 
narratives. The instrument consists of four scales that evaluate the examination, instructional competence, interestingness, 
excessive demand, learning and general estimation. Two scales of seven items were added to the original questionnaire to 
measure the accommodation resistance over time (non-solvable contradications, and solvable contradications). To rate the 
narrative elements, 5 items to rate the usability where added after the stories had been introduced. All items were measured 
on 7-point Likert-scales. 

The full evaluation study was conducted with 29 (5 male 24 female) master students and alumni who had taken a 
class before, which also used the L-MoSim cases in the same way as the class from the pilot study. We initially send out 45 
questionnaires. The return-rate was 64%. All the participants had completed the class successfully in the past. Their mean 
age was 24.51 years (SD = 2.35). At the time of evaluation, they did not currently work on the cases like the participants of 
the pilot study. They also received the same case both with and without the stories in it. For the new scale on 
accommodation resistance a =.69 can be reported. For the usability scale the a- value was a =.67. 

Results (Pilot Study) 

Table 2 shows the evaluation results of the pilot study between the first case without stories and the second case 
that included the newly developed biographical stories. 
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Table 2: Evaluation of the two different case-versions (pilot study) 



No stories (A) 

Stories (B) 

Scale 

M 

SD 

M 

SD 

examination 

6.57 

0.76 

6.71 

0.61 

Instructional competence 

4.53 

1.38 

5.46 

1.27 

interestingness 

5.56 

1.89 

5.57 

2.06 

excessive demand (turned) 

4.38 

1.88 

4.81 

1.29 

learning 

6.24 

0.77 

6.71 

0.46 

general estimation 

6.57 

0.79 

6.71 

0.49 

AR: non-solvable contradications 

2.65 

1.95 

2.20 

1.24 

AR: solvable contradications 

5.10 

1.81 

5.24 

1.76 


On a descriptive level, the second case rated higher on all scales but interestingness. The accommodation resistance 
dropped over time, which can be seen as a first indicator that stories help to strengthen the plausibility of the L-LoSim 
cases. To have a look at the usability of the stories within the cases 5 items were introduced. 

Table 3: Usability of the embedded stories (pilot study) 


Item 

M 

SD 

The stories that are embedded into the cases seem plausible to me 

6.57 

0.76 

The stories help me to understand the current situation of the virtual learner 

4.53 

1.38 

The stories help me to understand the case 

5.56 

1.89 

The stories help me to solve the case 

4.38 

1.88 

The stories help to understand the theoretical aspects of the personality traits 

6.24 

0.77 


Overall the usability was high. As regards the current situation and the solution of the simulated learner, other 
aspects of the case help probably more than the stories. The stories particularly helped with understanding the case and the 
underlying theory. The latter is interesting: initially, we introduced the stories to give more texture but they were 
constructed strictly on the basis of the theories of each individual personality trait. This deductive design-strategy obviously 
helped to understand the theory better - even though the theory was not explicitly named or described in the stories. The 
main evaluation study showed very similar results. 

Results (Main Evaluation Study) 

The main evaluation study had participants who reviewed the cases with and without the stories. All of the 
participants had successfully completed a class that used the cases as central and graded assignments. 
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Table 4: Evaluation of the two different case-versions (main evaluation study) 


No stories (A) Stories (B) T-Test and Effects 


Scale 

M 

SD 

M 

SD 



examination* 

5,79 

1,00 

6,58 

0,58 

t= 

-4.48, df=28, p<0.001, d=1.98 

Instructional competence* 

3,96 

1,52 

5,73 

0,95 

t= 

-10.663, df=28, p<0.001, d=1.98 

interestingness* 

4,96 

0,83 

5,57 

0,82 

t= 

-4.733, df=28, pcO.001, d=0.879 

excessive demand (turned)* 

4,28 

1,14 

5,14 

0,85 

t= 

-6.255, df=28, pcO.001, d=1.161 

learning* 

6,24 

0,55 

6,56 

0,43 

t= 

-2.88, df=28, p=0.008, d=0.535 

general estimation 

6,32 

0,62 

6,50 

0,34 

t= 

-1.837, df=28, p=0.077, d=0.341 

AR: non-solvable contradications* 

3,44 

0,73 

1,83 

0,71 

t= 

19.993, df=28, pcO.001, d=3.713 

AR: solvable contradications* 

4,13 

0,75 

5,39 

0,77 

t= 

-11.793. df=28, p<0.001, d=2.19 


* statistically significant against a type 1 error of at least p<0.05. 


Table 4 shows the effect in favor of the embedded stories present in the case for all scales but general estimation. 
This may be a type II error but this is not decidable on that sample size. The effect itself would still be considerable. The 
results of the evaluation study point into the same direction as the pilot study did. They also show a clear shift from the non- 
solvable-contradictions to solvable contradictions. We do not interpret this as a raise in self-efficacy in general - although 
we did not control for that - but rather for the effect of the additional texture. We conclude that the cases feel more real 
since the stories are part of the case. 

Table 5: Usability of the embedded stories (main evaluation study) 


Item 

M 

SD 

The stories that are embedded into the cases seem plausible to me 

5,90 

0,72 

The stories help me to understand the current situation of the virtual learner 

4,59 

1,12 

The stories help me to understand the case 

4,69 

1,69 

The stories help me to solve the case 

5,00 

1,16 

The stories help to understand the theoretical aspects of the personality traits 

6,34 

0,72 


Table 5 shows again the impressions of the stories within the case as regards usability. All items were rated very 
high. Again, both the plausibility and the theoretical understanding received the highest ratings. Raising plausibility was an 
intended goal of the story implementation. Initially, we did not predict the stories to help with the theory in particular. The 
availability of the stories also helps with understanding and solving the case. 

Discussion 

L-Mosim automatically creates very complex cases and tasks with a verbal surface that resembles real case 
histories of learners. The software aims at helping students with their studies on learning and on how to make complex 
decisions when individually coaching learners in their later professional lifes. L-MoSim creates a verbal document that 
contains notes by a simulated interviewer, original quotes by the simulated learner and now also stories from the past of the 
simulated learner that project selected variables to his or her past. It does so by processing a spreading activation simulation 
model that is based on real research data. This model is then transferred to text, and the numbers of the model nor its formal 
relations are hidden from the learner to create a more real learning experience. 

The results from the pilot study were overall promising towards embedding the biographical stories into the L- 
MoSim Software. The usability of the embedded stories indicated that the stories helped with the plausibility, with 
understanding the case itself and with the understanding the theories behind the simulation model. The results of the pilot 
study led to the main study. 
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The main evaluation study used the same instruments but addressed a larger sample size. The differences are even 
higher in the main study and high individual effects on all scales could be found. However, during the time of evaluation, 
the participants did not currently work on the cases. Yet, all of them used to work with the cases of L-MoSim during a past 
class, which they successfully completed. This may cause an interpretation problem. But on the other hand, the data of the 
pilot study and the main evaluation study show the same results. 

Although the stories added about 2 to 3 more pages to the whole case and task document, the users appreciated the 
additional content and texture. Initially we only introduced the stories as texture. The results however indicate that presence 
of the stories does more than just adding texture. It helps the user to understand the theoretical concepts better thus 
supporting the main learning goal of the simulation directly. 
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Abstract 

Mobile phones are used around the world, however, there are no definitive rules regarding their appropriate use. In 
the absence of such codification, the researchers surveyed 225 American and 1 15 Chinese college students to 
understand their perceptions of etiquette and appropriate use of mobile phones both in the classroom and in other 
public places. Several cultural similarities and significant mean differences were found. Similarly, results evince the 
need for established rules for mobile phone etiquette and the need to implement mobile phones in the curriculum. 

Descriptors: etiquette survey, mobile phone 


Introduction 

Mobile phones (alternately referred to as cell phones and in this paper include smart phone technology) 
proliferate in our society, influencing the way we communicate on a daily basis. They have been instrumental in the 
organization of “flash mobs” and other gatherings, both in the U.S. and abroad, and one might argue that they played 
an important role in the recent civilian unrest in Egypt and other countries, leading to changes in government 
(Sutter, 2011; Tarkowski, Fathy, & Melyantsou, 201 1). One cannot help but notice the many people who walk 
around any city or town holding or talking on a mobile phone. We can email, text, find information, use maps, take 
pictures, and even make phone calls on this amazing device. Considering the practicality and popularity of this 
communication tool, it is surprising that appropriate and polite use of the mobile phone is still unclear as there is no 
definitive set of rules for its use (Elgan, 2010; Rosenfeld & O'Connor-Petruso, 2010). Although 10 states have 
banned holding a mobile phone while driving and 35 states have banned text messaging while driving (Insurance 
Institute for Highway Safety, 2012), these are issues of safety rather than of etiquette. 

There is controversy related to mobile phone use in the classroom. Some people feel that mobile devices 
should not be allowed in the classroom because they can cause distractions both for the user and for those around the 
user and because they can be used for cheating (Tindell & Bohlander, 2012). The ringing of the phone and texting 
are two distractions that are sometimes mentioned. Interestingly, Charles (2012) found in her study that the latest 
generation of learners used the technology in school regardless of any rules to guide mobile phone use. Tindell and 
Bohlander report that many students perceive that their instructors are unaware of their texting during class. In a 
study of nursing students in the People’s Republic of China, Clark et al. (2012) found that cell phone use during 
class, including texting and making phone calls, was considered uncivil behavior. 

Geist (2011) considers that schools often ban mobile devices that can aid personal learning and social 
networking in favor of computers that provide learning in an outdated, more pedagogical way. However, Geist also 
found resistance to the use of mobile devices because some faculty thought that they were distractions that kept 
students from paying attention. Echeverria et al. (2012), Begum (201 1), and Geist point out that mobile devices have 
great potential for classroom use, but Geist cautions that many faculty members need to change their attitudes to 
accommodate the newest generation of learners. 

One can find web sites and other resources with suggestions for appropriate cell phone use, but rules for 
appropriate use have not been codified (Charles, 2012; Elgan, 2010). Charles reports that although some high 
schools set rules for appropriate use of mobile digital technology, these rules are not always clearly defined nor are 
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they enforced evenly. In her research, Charles notes that younger teachers often were more forgiving and flexible 
with rule enforcement than their older colleagues. 

In the college classroom professors often make their own rules about what is permitted and what is not; 
some professors allow the use of mobile phones and others do not permit phones to be out/on. Some professors have 
rules that seem to fall in between - students can use mobile phones only in certain situations that will benefit the 
class, for example, to look up something that is related to the class discussion (O’Connor-Petruso & Rosenfeld, 

201 1). Alobiedat (2012) observes that attitudes about mobile phones in the classroom may be influenced by the 
degree of the user’s experience with the technology. Baker, Lusk, and Neuhauser (2012) report that students in their 
study exhibited greater acceptance of in-class technology use than did the faculty, perhaps highlighting a difference 
between digital natives (students under 25) and digital immigrants (teachers). Some researchers (Baker et ah, 2012; 
Tindell & Bohlander, 2012) urge faculty members to include their policy on mobile phone use in their syllabi. 
Tindell and Bohlander state that if any policy is to be effective, it must be enforced. 

In a survey on mobile phone use of 487 inhabitants of Beijing, China, almost 90% of the respondents said 
that they turned off or set their mobile phones to vibrate in public places such as hospitals, movies, and classrooms 
(Fortunati, Manganelli, Law, & Yang, 2010). Interestingly, the researchers found that the primary reason for 
silencing their phones was not out of concern for others, but rather because of the limited duration of the phone 
battery. When respondents were asked what they do when their phone rings when they are in conversation with 
another person, over 80% apologize to the other person; almost half apologize and return the call later; one third 
apologize and answer the phone; less than 10% ignore the call and return the call later; and 2.7% stop the 
conversation and take the call. Fortunati et al. suggest that the Chinese have respect for the caller that is not 
generally found in western societies. Fortunati et al. indicate that this may be due to the traditional Chinese culture’s 
negativity toward the individual and privacy since any individual or private decision is considered as a selfish or 
improper desire. When respondents were asked about people who speak loudly on their mobile phones in public, 
80.9% declared that this behavior is impolite; 8.0% were indifferent; and 4.1% considered it normal. 

To try to better understand student perceptions about mobile phone use since there is no definitive set of 
rules, the authors previously surveyed 129 college students about their perceptions of appropriate and inappropriate 
uses of technology in the classroom and public spaces. They found no significant mean group differences among 
gender, culture, ethnicity and mobile phone use in the classroom or public spaces; however, they did find a 
generational theme. The younger age group (20-29 years old) supported texting in class and public places when they 
were bored and they were more inclined to use mobile phones for emergencies and in particular situations, 
dependent on the company they kept (Rosenfeld & O’Connor-Petruso, 2010). 

The first survey was subsequently refined to develop a second survey that honed in on perceptions of 
appropriate mobile phone use. During this second round of research, the authors surveyed 316 college students and 
found that all ethnic groups represented seemed to believe that it is socially unacceptable or inappropriate to make 
phone calls or text during university lectures, university class discussions, and group work. All ethnic groups 
perceived it is socially acceptable or appropriate to text in restaurants when alone, and make and take cell phone 
calls in public places such as doctor's waiting rooms, libraries, museums and on public transportation (O’Connor- 
Petruso & Rosenfeld, 2011). 

The authors found that younger students tend to text more per day, and text more during both university 
lectures and class discussions. Younger students text and miss more information from university lectures and 
cooperative work, they have a tendency to text while driving, and they believe their quality of life/social life has 
greatly been enhanced due to texting and visiting or posting to social networking sites (O’Connor-Petruso & 
Rosenfeld, 2011). 

The present study uses the same survey instrument to examine the perceptions of 340 American and 
Chinese college students to see if there are any differences in their perceptions of appropriate mobile phone use. The 
present research questions are: 

Hri Are there differences in the frequency of mobile phone use between American and Chinese college students? 
Hr 2 Are there differences in the perceptions of mobile phone use of American and Chinese college students with 
regard to gender, age, or ethnicity? 


Methods 


Participants 

The present survey consists of 225 (66%) American college students and 1 15 (34%) Chinese college 
students. Sixteen percent (n= 35) of the American students are male and 84% (n=190) are female. Thirty-four 
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percent ( n=39) of the Chinese students are male and 66% (n=76) are female. The age range of the American college 
students spans five decades (13-60 + years old) with the largest number of participants between 20-29 years old 
(62%), followed by 30-39 years old (18%), and 40-49 years old ( 11%). The age range of the Chinese students spans 
three decades (13-39 years old) with the largest number of participants between 20-29 years old (85%) and 13-19 
years old (10%). 

Instruments 

The self-administered questionnaire used for the analysis was the International Technology Etiquette 
Survey, v. 3 (O'Connor-Petruso, Rosenfeld. Martinez-Pons, 2011). This survey instrument was pilot tested and has a 
Cronbach's alpha reliability coefficient of .791. The survey consists of 60 questions and is divided into three parts: 
Part I) Demographics & Background Information, Part II) Frequencies, and Part III) Agree/Disagree (set to a Likert 
Scale as follows: 1. Strongly Disagree, 2. Disagree, 3. Agree, and 4. Strongly Agree). This survey has been 
translated into other languages and is presently available on the Web. 

Procedure 

All participants were asked to take the online survey (located at http://internationaltechnologystudies.org) 
and select either the English version or the Chinese version (Wei (translator), 2011) respectively. Data were 
collected over a ten month period. 


Analysis 

The research data was analyzed using IBM's PASW (Predictive Analytics Software), v. 19. Descriptive 
statistics and t-tests were run to ascertain frequencies and mean differences. Results are reported as well as trends 
that were found. 


Results 


Frequencies 

One hundred percent of the participants use a mobile phone. A majority of both the American students 
(84%) and Chinese students (55%) use a smart phone, make/take 1-6 cell phone calls per day and text everyday. 
However, the number of texts per day varies between cultures. Forty percent (n=91) of American students average 
21+ texts per day, and 31% (n=70) average 4-10 texts per day. Six percent (n=7) of the Chinese students average 
21+ texts per day, and 62% (n=72) average 4-10 texts per day. The medium of choice for local communication for 
both cultures is the cell phone, followed by e-mail and texting. More than three-fourths (77%; n=174) of the 
American students prefer cell phone calls, 59% (n=132) prefer email and 35% (n=78) prefer text. More than two- 
thirds (70%; n=80) of the Chinese students prefer cell phone calls, 52% (n=60) prefer e-mail and 48% (n=55) prefer 
text. 

More than one -third of all participants with children had their own mobile phones. Forty-eight percent 
(n=34/78) of American participants and forty percent (n=6/15) of Chinese participants gave their child/children their 
own cell phone. 

More American students were active on Skype and Social Networking Sites (SNS) than the Chinese 
students. Thirty-eight percent (n=85) of American participants have active Skype accounts, 84% (n=189) belong to a 
SNS, and 77% (n=174) are active at least once or twice per week. Eighteen percent (n=21) of Chinese participants 
have an active Skype account, fifty-five percent (n=63) belong to a SNS, and 72% (n=83) are active at least once or 
twice per week. 

The majority of all participants believe it is socially acceptable to text in restaurants when alone - American 
students (77%; n=173) and Chinese students (88%; n=101); with friends and/or colleagues - American students 
(61%; n=137) and Chinese students (77%; n=89); and in a doctor’s waiting room - American students (72%; n=163) 
and Chinese students (81%; n=93). More than one-half of all participants believe their quality of life (especially 
their social life) has greatly increased due to texting and/or posting on social networking sites - American students 
(51%; n=114) and Chinese students (56%; n=64). 

Concerning university class lectures, more than two-thirds of both American students (74%; n=160) and 
Chinese students (74%; n=85) believe it is inappropriate to make or take cell phone calls even if you speak in a low 
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tone. However, fifty percent or more of both American and Chinese students believe is socially acceptable to text 
during university lectures - American students (67%; n=150) and Chinese students (50%; n=57); and during 
university class discussions - American students (67%; n = 150) and Chinese students (60%; n=69). Although the 
majority of the participants admit to missing information from lectures when texting - American students (69%; n = 
155) and Chinese students (77%; n=88), both groups believe they can text and make and take cells phone calls and 
not miss any information when working cooperatively with colleagues - American students (67%; n = 150) and 
Chinese students (60%; n=69). 

More than three-fourths of both American and Chinese students believe cell phones should be on vibrate 
mode during university lectures - American students (84%; n=189) and Chinese students (91%; n=105), and class 
discussions and/or group work - American students (66%; n=148) and Chinese students (87%; n= 100). 

The majority of all participants believe it is socially acceptable/appropriate to make/take cell phone calls in 
restaurants - American students (64%; n=144) and Chinese students (87%; n= 102), in a doctor’s waiting room even 
if other people are present - American students (51%; n=l 15) and Chinese students (70%; n= 61), and public places 
such as libraries, museums, and/or using public transportation - American students (56%; n=123) and Chinese 
students (63%; n= 73). 

The majority of all participants also believe they are very courteous and speak in soft tones when using 
their cell phone - American students (89%; n=200) and Chinese students (91%; n= 1 15); and that most people are 
discourteous and speak in loud tones - American students (71%; n=159) and Chinese students (67%; n= 58). More 
than two-thirds of both groups of students believe there should be rules/established etiquette for where and when to 
use cell phones - American students (72%; n=161) and Chinese students (77%; n= 89). 

More than one-half of all the participants wish the educational system would create instructional strategies 
that utilize cell and/or smart phones - American students (57%; n = 128) and Chinese students (62%; n=71). 
Approximately two-fifths of all participants believe texting has increased their ability to think and/or problem solve 
- American students (40%; n = 91) and Chinese students (38%; n=44). 

The majority of the participants in both groups state they are not “multi-taskers” and cannot easily text 
and/or make cell phone calls in any environment - American students (53%; n = 119) and Chinese students (62%; 
n=71). Fifteen percent (n=34) of American students and 11% (n=13) of Chinese students report they frequently text 
while driving a car. 

t-Tests 


Independent samples t-Tests reveal several mean cultural differences. Effect size statistics are reported 
(using guidelines proposed by Cohen, 1988). Eta squared statistics reveal small but significant differences. 

More American students believe it is socially acceptable to text during university class lectures (t(338) = - 
3.260, p<.05) than Chinese students. The mean of the American group was significantly higher (m =2.83, sd = .967) 
than the mean of the Chinese group (m = 2.55, sd = .625). Eta squared is .03. Sixty-seven percent (124/190 females 
and 26/35 males) of American students as compared to sixty-percent (48/76 females and 21/39 males) of Chinese 
students believe texting during class lecture is appropriate - there were no significant differences between gender 
groups. 

More Chinese students believe cell phones should be turned off during university lecture (t(338) = 3.263, 
p<.05) than American students. The mean of the Chinese group was significantly higher (m =2.56, sd = .703) than 
the mean of the American group (m = 2.24, sd = .927). Eta squared is .03. Fifty-four percent (45/76 females and 
17/39 males) of Chinese students as compared to thirty-two percent (61/190 females and 1 l/35males) favor turning 
cell phones off during lecture - there were no differences between gender groups. 

More Chinese students also believe cell phones should be turned off during university class discussions 
and/or group work (t(338) = 3.334 , p<.05) than American students. The mean of the Chinese group was 
significantly higher (m =2.52, sd = .705) than the mean of the American group (m = 2.22, sd = .918). Eta squared is 
.03. Fifty-six percent (56/76 females and 21/39 males) of Chinese students as compared to thirty-two percent 
(61/190 females and 12/35males) favor turning cell phones off during lecture -there were no significant differences 
between gender groups. 

More Chinese students believe they can easily text and/or make and take cell phone calls while working 
cooperatively and take cell phone calls while working cooperatively with colleagues - without missing any 
information (t(321) = 2.539 , p<.05) than American students. The mean of the Chinese group was significantly 
higher (m =3.03, sd = .599) than the mean of the American group (m = 2.78, sd = .836). Eta squared is .02. Eighty- 
seven percent (72/76 females and 30/39 males) of Chinese students as compared to sixty-six percent (121/190 
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females and 27/35 males) of American students perceive they can multi-task on the mobile phone and concentrate 
on group work without missing information -there were no significant differences between gender groups. 

More American students believe texting during class lectures, discussions, and/or group work helps relieve 
boredom (t(31 1) = -2.877, p<.05) than Chinese students. The mean of the American group was significantly higher 
(m =2.58, sd = .984) than the mean of the Chinese group (m = 2.32, sd = .669). Eta squared is .02. Fifty-four percent 
(103/190 females and 19/35males) of American students as compared to forty-two percent (31/76 females and 17/39 
males) of Chinese students believe texting during various class activities relieves boredom - there were no 
significant differences between gender groups. 

More American students also believe texting during meetings helps relieve boredom (t(396) = -2.519, 
p<.05) than Chinese students. The mean of the American group was significantly higher (m =2.72, sd = .970) than 
the mean of the Chinese group (m = 2.49, sd = .707). Eta squared is .02. Sixty percent (114/190 females and 20/35 
males) of American students as compared to fifty-four percent (46/76 females and 16/39 males) of Chinese students 
believe texting during various class activities relieves boredom - there were no significant differences between 
gender groups. 

More American students believe most people are discourteous and talk in loud tones when using cell 
phones (t(338) =-3.129, p<.05) than Chinese students. The mean of the American group was significantly higher (m 
=2.89, sd = .658) than the mean of the Chinese group (m = 2.61, sd = .7854). Eta squared is .03. Seventy-one 
percent (159/190 females and 21/35males) of American students as compared to fifty-eight percent (49/76 females 
and 18/39 males) of Chinese students believe mobile users are rude and loud - there were no significant differences 
between gender groups. 


Discussion 

In response to H R i, there are more similarities than differences between American and Chinese students 
concerning the frequency of mobile phone use. All the survey participants use the cell phone every day to either 
make or take cell phone calls and text; however, more American students than Chinese students send more texts per 
day. The medium of choice for both groups of students is the mobile phone. 

Although more American students were active on Skype and SNS than their colleagues in the East, Chinese 
students are also active on these Web 2.0 tools. The majority of the participants from both groups believe their 
quality of life has increased due to texting and being active on SNS. 

In response to H R2 , there are again more similarities than differences between American and Chinese 
students concerning their perceptions of mobile use; however, there are small but significant mean differences 
between the cultures in some areas of texting and mobile phone behaviors. 

The majority of both the American and Chinese students believe it is socially acceptable/appropriate to text 
in university class lectures and discussions, restaurants, doctors’ offices, and public places like libraries and 
museums, and public transportation. Both groups also state they are not multi-taskers and miss information from 
university lectures when texting; in contrast however, both groups believe they are able to concentrate on 
collaborative university assignments while simultaneously texting. 

Although the majority of both groups believe it is inappropriate to make or take cell phone calls during 
university lectures, the majority of the participants believe it is socially acceptable to make or take cell phone calls 
in restaurants, doctors’ waiting rooms, public places such as libraries and museums, and public transportation. 

More than 50% of both the American and Chinese student wish the educational system would implement 
instructional strategies that utilize their mobile phone and approximately 40% of both groups believe texting has 
increased their cognitive abilities. 

The majority of American and Chinese participants believe they are courteous cell phone users and speak 
in soft tones while the majority of other mobile phone users are discourteous and speak in loud tones. Both groups 
overwhelmingly decry the need for rules/established etiquette on mobile phone use. 

Small but significant mean cultural differences were evinced in the following areas: More American 
students than Chinese students believe a) texting is socially acceptable in university lectures, b) texting helps relieve 
boredom from university class lectures, discussions and/or group work, and c) most people are discourteous and 
speak loudly when using cell phones. More Chinese students than American students believe a) cell phones should 
be turned off during university lectures and class discussions and/or group work, and b) they can easily text and/or 
make cell phone calls while working cooperatively with colleagues and not miss any work. 
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Implications 


The results of this study clearly show an increase in the level of mobile phone use from 201 1 to 2012 in 
diversified environments, causing aggravation, deterring learning, and threatening safety conditions to all (texting 
while driving). 

The authors propose “Three Simple Guidelines for Cell Phone Use” (O’Connor-Petruso & Rosenfeld, 

2011): 

1) Cell phones should be turned off/vibrate mode in the classroom and public places. 

2) Texting should not be permitted during university lecture/discussion/meeting unless it 
is part of the curriculum/agenda. 

3) If possible, one should always remove themselves from public places when 
making/taking cell phone calls. 

In agreement with prior research (O’Connor-Petruso, 2011; Rosenfeld & O’Connor-Petruso, 2010), the 
results categorically evince the need for curricular change - specifically the request of college students from both 
sides of the hemisphere to implement cellular technology into university assignments and activities. 

The authors also note the need for more research from male participants, older age groups, and other 
cultures in order to query gender differences, and ascertain if these findings are generational (blame it on the 
younger generation) or endemic due to the Information Age. 
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Abstract 

A training evaluation project conducted in an international company is introduced in this paper. The authors give 
an overview of the evaluation approaches employed in the project and the data collection methods including 
training evaluation forms, online test, online survey and interviews. The results of data analysis indicated that the 
training was not successful and did not meet the needs of the training participants and the organization. The authors 
discuss the problems existing in the training program that impeded performance improvement, and recommend the 
ways of linking a training intervention to performance improvement by reflecting on the evaluation process. 

Introduction 

XYZ is an international company in the chemical industry with the headquarter located in the Indiana State 
and its manufacturing and commercial facilities scattered throughout the United States, Europe, and Asia. An eight- 
hour training workshop was administered right after a company sponsored three-day technology conference in 201 1. 
The goal of the workshop was to improve the thirty trainees’ experiment-design knowledge in order to increase their 
skill level in the areas of research, development, and testing projects. An external trainer from a training vendor 
facilitated the training workshop. In this evaluation project, the Kirkpatrick’s evaluation process was employed to 
help answer the following questions about the program with regards to necessity, quality and value: 

• Did the training meet participants’ learning needs? 

• Did the trainees feel more confident in designing experiments? 

• Have trainees applied what they learned to their jobs? 

• Are trainees able to design experiments that lead to better results in a shorter time? 

Evaluation Methods 

Several methods were used to evaluate the training workshop, drawing on Kirkpatrick’s 4 Levels of 
Evaluation as the primary evaluation approach. The Kirkpatrick method of evaluation is a well-established process 
and has been used in business to help evaluate the effectiveness of training programs. The 4 levels in order are: 
Reaction, Learning, Behavior, and Results. The section below is a breakdown of the approaches used at each level. 
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Level 1: This level evaluated how well the training was received. Did the instructor deliver the training in 
an understandable way? Were the trainees engaged in the learning process? Were the course materials and facilities 
conducive to their learning? The data for this level was collected after the conclusion of the training using an 
evaluation form developed by the training vendor who was contracted to deliver the workshop. Feedback from 24 
trainees was collected. 

Level 2: At the time of the training, no formal testing was conducted. With the help of the trainer, a test 
was designed around the objectives of the training and administered 2 months after the training delivery. The 
purpose of the test was to assess the level of learning retained since the training took place. The test required trainees 
to apply what they had learned in the training. It contained a set of simple but well-defined problems that could only 
be answered if the participant had retained what was learned. The test was administered online using the XYZ 
Company’s Learning Management System (LMS). Reminders were sent to the training participants twice and by the 
deadline seven completed tests were received. 

Level 5: A transfer evaluation was conducted in order to identify to what degree participants applied what 
they learned in the training as well as to ascertain what factors, if any, impeded the transfer of the learning. The key 
transfer evaluation questions were: (1) To what degree have the training participants applied what they learned in the 
workshop on their job? (2) If training transfer did not occur, what factors impeded the transfer process? (3) How 
confident/committed are the training participants that they will be able to apply what they learned in the workshop 
on their job in the future? Interviews were conducted 2 months after the training to access the change in behavior of 
the participants. Data was collected using the following methods: 

• Interview with training participants 

A semi-structured interview protocol was designed to learn to what degree the training participants applied 
what they learned during the training when they were back on the job, as well as the factors leading to the degree of 
training transfer. 

• Interview with supervisors of training participants 

A semi-structured interview protocol was designed to learn whether participants demonstrated behavioral 
changes and what changes their supervisors have observed or heard. 

• Interview with the trainer 

A structured interview protocol was designed to learn what the trainer would expect training participants to 
transfer to their job after the workshop. 

• Interview with the Training Manager 

A structured interview protocol was designed to learn further detailed information about the background of 
this training workshop. 

• Online survey 

Based on preliminary feedback from the interviews, an online survey was developed to collect more 
feedback from the training participants, with respect to transfer of skills learned in the specific training workshop. 
The online survey was setup so that participants’ responses were anonymous. The response rate was up to 30.4% by 
the proposed deadline. 

Level 4\ Interviews with supervisors and the training manager were conducted to access the impact of the 
training and to what degree the training allowed the training participants to optimize their testing processes. The 
relevant data were collected through the interviews and online survey mentioned above. Questions were asked to 
find out whether the desired results were achieved and if not now, the possibility of achieving desired results in the 
future, such as "do you think you have reduced the time of your testing processes after the training", "have you been 
able to conduct tests more effectively in the lab", “how confident are you that you will be able to apply what you 
have learned in the workshop on the job in the future”, and “how committed are you to applying what you have 
learned in the workshop on the job in the future”. 


Results 

Level 1: Reaction to the training program: “To what degree did training participants react favorably to the training 
workshop? ” 

The data collected from the initial evaluation forms showed that the trainees thought that the instructor was 
knowledgeable in the subject area. According to the participant data, the instructor was able to clearly teach the 
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training content, and attempted to interact with the trainees. Another problem was that the training didn’t teach 
people how to use a specific statistics software ABC introduced in the company. The problem could have been 
avoided through a thorough needs analysis at the very beginning. With regards to the training strategy, it would be 
better and more helpful to present some real-life examples. See Table 1 below. 

Table 1 shows a summary of the data collected from the level 1 evaluation 


Evaluation Item 

Mean 

Median 

SD 

Instructor’s knowledge of the subject 

4.46 

4.5 

.59 

Instructor’s presentation/facilitation skills 

3.67 

4 

.92 

Course materials 

3.54 

4 

.93 

Training environment 

3.74 

4 

.86 

Overall effectiveness 

3.79 

4 

.72 


The feedback from the open-ended questions included: 

( 1 ) Instructor presentation/facilitation skills: the general reaction was that the instructor was able to clearly 
teach the training content. The data also supported this instructor’s strength in this area. He received an average of 
4.5 with a low standard deviation. This would indicate agreement with the mean score. One trainee commented that 
“the instructor did a good thing trying to involve and include people”, and also suggested the instructor “explain the 
computer program more for people who have never used it or don't know anything about it”. With regards to the 
training techniques, the feedback indicated that it would be more practical and helpful to present real-life examples 
that demonstrated to use the statistics software. 

(2) Course workbook: the mean score in this area was 3.5 with a standard deviation of .90. This would 
indicate that the data was more widely dispersed. One trainee stated that it “jumped a little quickly from the basics 
to the more advanced” while another commented that the material was “a little confusing” but overall it was “well 
presented and divided into easily understood sections”. The different reactions may not be difficult to understand as 
the training participants had different levels of subject knowledge and skills. 

(3) Suggestions: some trainees said that they would like to “see more on how to use the statistics software 
ABC”. This revealed that the training didn’t teach the training participants what they were expecting to learn, in 
other words, the training didn’t meet some trainees’ learning needs. 

Level 2: Learning: “To what degree did training participants acquired the intended knowledge and skills based on 
their participation in the training workshop?’’ 

The company’s LMS records indicated that 7 of the 23 training participants were able to complete the 
online test and their scores were recorded in the LMS. The status of the other 16 participants showed “in progress”. 
This means either they didn’t complete the online test or their grades were below 60 (out of 100). In addition, the 
tenure information of the seven trainees who completed the test was identified and mapped with their grades. There 
did not seem to be any correlation between length in job and test performance. The mean test score was 80% with a 
standard deviation of +/- 12.8%. This reflects the large gap between scores, which ranged between 90% and 60%. 
Given that only 7 test scores were tabulated the data is not wholly reliable. 

Level 3: Behavior: 

“To what degree did training participants apply what they learned during training when they were back on the job? 

The Training Manager was expecting the training participants to perform better in the lab after attending 
the training workshop. In the online survey in which the response scale ranged from 0 (not at all) to 10 (very much), 
7 people responded to the question, regarding their ability to apply what was learned. Among the respondents, 57% 
of the respondents checked 0 or 1. The mean response was 3.1 with a standard deviation of +/-3.1. However, with 
the low response rate of only 7, this data is not wholly reliable. 
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If training transfer did not occur, what factors impeded the transfer process? 

It is important to identify the reasons why training transfer did not occur as expected. Several problems 
were identified by analyzing the survey data. Sixty percent of the respondents selected the reason for transfer failure 
as “I am not required to do this”. Other selected reasons were that “I do not have the necessary knowledge and 
skills”, “I do not have the necessary resources to do it”, and “I have other higher priorities”. Other feedback included 
that “the content of the training was not pertinent to the current project status”, and “nothing new was learned in the 
workshop since several courses had been taken previously”. 

Specifically, one interviewee was responsible for the quality control labs and creating new methods to 
support research and development. She said that the training was assigned to her, not requested based on her needs. 
Before the training she had very little experience in Experiments Design and had a personal interest in the training 
workshop however the workshop didn't teach her well enough to transfer the skills. In addition the statistical 
software presented in the training workshop was not the one used within the company. It is also worth noting that 
participants were not provided with computers to use during the training. Even if the correct software was used, the 
lack of computers to work with might have hindered the learning. The interviewee said that she was confused as 
steps using that particular software were quite different from what she was using in the company and as a result she 
wouldn’t have been able to apply the steps on her job after the training. She also commented that no certification 
was granted at the end of the workshop. 

Level 4: Results: “How confident/committed are the training participants that they will be able to apply what they 
learned in the workshop on their job in the future?” 

The data shows that 71% of the respondents reported their confidence level was below 6 (0-10, 0: not 
confident at all; 10; extremely confident). The mean response was 5.00 with a standard deviation of +1-3.9. 

Similarly, 71% reported their commitment level was below 6 (0-10, 0: not commitment at all; 10: extremely 
committed). The response mean was 3.714286 and the standard deviation +/- 4. 

Findings 

• Did the training meet participants’ learning needs? 

Given that there was no needs analysis done, it is not surprising that participant needs where not met 
because they were unknown. As a result, the training workshop was not customized to meet participants' training 
needs nor were the correct participants identified. The training workshop was not customized based on the statistical 
software used in this company; therefore it didn’t provide appropriate knowledge and skills needed by the training 
participants. Learning needs were further hindered because there were not computers in the classroom that would 
allow participants to interact with the software. 

• Did the trainees feel more confident in designing experiments? 

Most of the training participants showed a lack of confidence and commitment in applying what they 
learned in the workshop in their jobs. In order to aid training transfer in the future it is essential to resolve the 
problems identified above with this workshop. In addition, there were also issues on the organizational level that 
were not exclusive to this training. 

• Have trainees applied what they learned to their jobs? 

The training participants were not specifically required to apply what they learned on their job. The training 
participants were still able to perform their routine jobs without any training transfer. The training transfer did not 
occur as expected. 57% of the respondents never applied what they learned on the job or applied very little. 

• Are trainees able to design experiments that lead to better results in a shorter time? 

The overall pass rate on the test was good. It is unclear though if the pass rate can be directly attributable to 
the training since so few trainees finished the test. Those who passed might have had prior knowledge of the subject. 
Length in job in this sample does not seem to correlate to scores. Test scores may correlate more closely to job 
responsibilities and educational background. 
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• Was the result of streamlining research and development methods achieved? 

It was not possible to calculate ROI because all expense figures were not available. It is fair to conclude 
that the ROI was low, as the training was essentially not transferred. The XYZ Company intended this training to 
help facilitate a positive change in how experiments were conducted in their labs. The company leadership wanted 
to streamline research and development methods in lab testing. The objective for offering this training was to give 
lab employees a tool for reducing and minimizing the tests that have to be performed while still producing reliable 
and useful data. Unfortunately, we were not able to find evidence that participants were using this training. There 
did not appear to be a clear link between the Company's goals and training needs. 

Limitations 

There were some limitations during our data collection and analysis process. First, time was limited. 
Second, the training group was relatively small in size. As a result, we received only seven responses from the 
training participants. However, comparing the response rates of other surveys conducted in this company (20%- 
35%), to this response rate (30.4%) we found that it was, in fact, an above average response rate. Thirdly, we were 
only given access to 10 of the training participants for interviews because of availability issues. It would have also 
been helpful if we could have had interviews with supervisors. Even with these limitations, we were still able to 
identify some important issues with the training by analyzing the currently available data. 

Recommendations 

Ways of linking a training intervention to performance improvement are recommended by reflecting on the 
training evaluation process: 

• Link company initiatives and goals to a training intervention. It will also aid in identifying the correct type 
of training content needed. 

• Conduct training needs analysis to ensure that the training will be appropriate and in context of the 
learners' needs as well as those of the organization. 

• Select training participants based on their job requirements and developmental needs. This will help ensure 
that the employees who need this type of training are appropriately identified. It will eliminate the waste of 
sending employees to a training session that they will not use. 

• Clarify with the selected participants why they have been asked to attend the training and what is expected 
of them after the training has been concluded. First, emphasis should be placed on why the training is 
important to the overall success of the company and then show participants how the training can help them 
do their jobs, better, faster and more efficiently. 

• If participants will be required to use a specific kind of software to perform tasks, ensure that the learning 
environment is equipped with computers and relevant software for all participants to participate in the 
training session. 

• Encourage supervisors to engage employees in discussions about how they will use what they learned on 
the job and the benefits of applying what was learned. 

• In addition to the training manual provided, it may be helpful to provide job aids if there will be a delay 
between when training is delivered and when the trainee will actually be using what has been learned. 
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Abstract 

Research indicates that there are not enough studies on the practices used by teacher education programs to 
prepare teachers to teach with technology or of their effects on teachers. As part of an ongoing multiple phase study, 
this study report the preliminary findings of how K-12 teachers were prepared versus their actual professional 
practices with technology integration. The perspectives of K-12 administrators and teacher education faculty are also 
analyzed for areas of agreement and disconnect. 


Introduction 

Research has indicated a lack of studies regarding what methods are used in teacher education program for 
conveying technology integration concepts and what affects these methods have on their students (Hew & Brush, 
2007; Pellegrino et al., 2007; Lawless & Pellegrino, 2007). While some research has been conducted on the use of 
different approaches and strategies in Schools of Education, the problem has been with the evaluation process not 
being completed in a careful and rigorous manner (Kay, 2006) or the results not being reported (Polly et al, 2010). 
The numbers of detailed study examples are limited (e.g. Strudler, McKinney, Jones, & Quinn, 1999) and concern 
has been expressed that there has been a limited number of cross-institutional studies (Pellegrino et al.). 

Purpose 

The purpose of this ongoing study is to examine differences in teacher technology preparation and usage. 
These differences in preparation are being explored through consideration of their subject matter area, grade level, 
and teacher education program. Differences are also being explored based on the perspectives of K-12 teachers, K- 
12 administrators, and teacher education faculty members to identify differences in approaches, values, and 
reasoning. 


Participants 

The participants in this study are drawn from three different populations: K-12 teachers, K-12 
administrators, and teacher education faculty members. The K-12 teachers being targeted are those whose main 
responsibility at their schools is to teach one of the core subject areas or if they are an elementary teacher, have a 
self-contained classroom (i.e. they work largely with the same students during the day). The K-12 administrators 
being targeted are any administrators with knowledge of their teachers’ practices with technology. The teacher 
education faculty members being targeted are those faculty members responsible for technology integration courses. 
In the event that a teacher education program does not require a standalone technology course, methods faculty 
members are being targeted to examine what technology content is addressed in those courses. The study setting is 
that of all of the K-12 schools and all of the universities and colleges in the same American South Eastern state that 
have a Teacher Education Program. Preliminary findings are reported for the K-12 teachers (n = 50), K-12 
administrators (n = 13), and teacher education faculty representing unique institutions (n = 16). Recruitment is 
ongoing. 
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Research Questions 


The research questions informing this study are: 

1) How does the technology integration training that teachers receive as part of their Teacher Education 
Programs compare with the technology integration practices of teachers in the field? 

2) What commonalities and differences exist in how Teacher Education Programs prepare their students to 
integrate technology into their classrooms and teaching? 

3) What commonalities and differences exist in how different secondary subject teachers integrate technology 
into their classrooms and teaching? 

4) What commonalities and differences exist in how different elementary teachers integrate technology into 
their classrooms and teaching? 

5) What commonalities and differences exist in how different secondary subject teachers integrate technology 
into their classrooms and teaching versus that of elementary teachers? 

6) What commonalities and differences exist between the perspectives of K-12 administrators regarding 
technology preparation and integration practices versus those of the K-12 teachers and teacher education 
program faculty? 


Method 

This study utilizes a mixed method research design (Merriam, 2009). Between the three phases of the 
study, each phase contains a different data collection method: survey research, semi-structured qualitative phone 
interviews, and ethnographic observations in K-12 and university classroom settings. Only the survey portion of the 
study is underway at present. 


Data Source 

The data sources consist of the online questionnaires, phone interview transcripts, and video recorded 
observations from site visits. Only initial findings from the survey responses are reported at this time. 

Procedure 

Participants were recruited in several ways. Recruitment messages were sent to listservs for K-12 teachers 
and administrators in the Southeastern state where this study was carried out. These same groups were given 
recruitment information during visits to a university for conferences and by working with the university’s teacher 
education program office to leverage contact information for schools that they work with to place student teachers. 
Teacher education faculty members were targeted for recruitment based on reviews of their university’s public 
websites regarding teacher education faculty. Only those faculty members responsible for technology integration 
courses were sent recruitment materials. In the event that there wasn’t such a course offered, department chairs and 
deans who could properly direct requests were contacted instead. Participants from each group are then able to opt 
into their respective interview phases by providing contact information at the end of the first phase survey. 
Participants from each group will then able to opt into the observation phase of the study by indicating an interest 
during the phone interviews from the second phase. Closed-ended survey responses were analyzed using descriptive 
statistics, while open-ended responses were analyzed using content analysis (Merriam, 2009). Interviews and 
observations have not been completed at this time. 

Selected Results 

Not all participants in each group responded to all of the survey questions; hence not all reported results 
will be out of the total number of participants for each group. Selected results reported at this time are from closed 
response questions on the survey. 
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K-12 Teachers 


A total of 65 teachers have opened the survey, most (n = 50/65) have provided enough responses to be 
considered for analysis. Of the teachers that have responded so far, most (n = 36/50) indicated that they taught 
grades six through eight. The next highest total was for grades three through five (n = 8/50) and equal results for 
Pre-Kindergarten through second grade and grades nine through twelve (n = 4/50 each). When asked how well 
prepared they felt they were to use technology in their teaching the majority of teacher felt that they were either 
“well prepared” (n = 25/48), “somewhat prepared” (n = 14/48), or “extremely well prepared” (n = 7/48). Only two 
teachers indicated that they felt they were not prepared. When asked if their teacher education program had specific 
technology requirements they needed to complete as part of their initial teacher certification program the majority (n 
= 29/47) indicated that they did. When asked to identify these technology requirements, most of the teachers 
identified either an educational technology class (n = 25/32), completing technology projects/activities in teaching 
methods courses (n = 24/29), or completing technology projects/activities in other educational courses (n = 20/29). 
In reporting average weekly technology use the most frequent uses were for administration and classroom 
management (n = 44/44), communication (n = 44/44), accessing and using electronic resources (n = 44/44), personal 
productivity (n = 42/44), classroom preparation (n = 41/44), and information presentation (n = 40/44). The least 
frequent uses were supporting higher order thinking skills (n = 18/44) and supporting students with special needs (n 
= 17/44) (See Figure 1 for additional activities). To date only a portion (n = 10/50) of the teachers have opted into 
the interview phase of this study. 

K-12 Administrators 

A total of 40 administrators have opened the survey, with few (n = 13) providing enough responses to 
analyze and fewer (n = 9/13) consistently responding to most of the questions. Of the administrators that have 
responded so far, most are principals at an elementary school (n = 9/13) serving either Pre-Kindergarten through 
second grade, grades three through five, or both. The remaining administrators are either in charge of a school 
serving grades six through eight (n = 2/13), grades nine through twelve (n =1/13), or serving all grades (n = 1/13). 
When asked how well prepared they felt their teachers were to use technology in their teaching, most of 
administrators indicated that they were either “well prepared” (n = 4/9), “somewhat prepared” (n = 4/9), or 
“extremely well prepared” (n = 1/9). No administrators that responded to this question indicated that that they felt 
their teachers were “not prepared." When asked if their teachers’ teacher education program had specific technology 
requirements they needed to complete as part of their initial teacher certification program the majority (n = 5/9) 
indicated that they did. Of the administrators that could comment on specific teacher technology experiences, all (n 
= 5/5) of them indicated that their teachers had a technology course, had to complete technology projects/activities 
in teaching methods courses and in other education courses, observe technology use during classroom observations, 
and develop/implement technology lessons/activities during both field experiences and student teaching. When 
asked to identify how their teachers used technology on a weekly basis most of the administrators who responded to 
this question indicated the most frequent uses were information presentation (n = 8/8), communication (n = 8/8), 
classroom preparation (n = 8/8), personal productivity (n = 7/8), administration and classroom management (n = 
7/8), accessing and using electronic resources (n = 7/8), and analyzing student data (n = 7/8) (See Figurel for 
additional activities). To date only a portion (n = 3/13) of the administrators have opted into the interview phase of 
the study. 

Teacher Education Program Faculty 

Of the colleges or universities with teacher education programs contacted (n = 44) a total of 20 teacher 
education faculty members have opened the survey, with most (n = 13/20) providing enough responses to analyze. 
When asked what kinds of technology experiences were required in all of their teacher education programs, most of 
the school indicated that they required an education technology course (n = 8/13), technology projects/activities in 
teaching methods courses (n = 8/13), and the development/implementation of technology lessons/activities during 
students teaching (n = 7/13). When reporting on technology courses that were required by all teacher education 
programs at the schools, the technology topics addressed by a majority of the schools included personal productivity 
(n = 13/13), information presentation (n = 13/13), communication (n =12/13), accessing electronic resources (n = 
12/13), administration and classroom management (n = 1 1/13), and documenting personal/professional growth (n = 
10/13). The least frequently addressed technology topic in technology classes required by all teacher education 
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programs in a school was using technology to facilitate teaching specific concepts (n = 6/13) (See Figure 1 for 
additional activities). To date only a portion (n = 7/20) of the faculty members responding on behalf of their 
university have opted into interview phase of this study. 



"Teachers 
1 Admins 
fl ANTE Programs 
■ Some TE Programs 


Figure 1. A comparison of weekly types of technology use reported by K-12 teachers and administrators as 
compared to the same technology uses identified as being present in “all” or “some” teacher education programs. 
*For teacher education programs special needs was subsumed by learning styles and classroom prep subsumed by 
personal productivity. 


Discussion 

This is an ongoing multi-year research study that was planned in Fall 2011 and undertaken in January of 
2012. The results reported above are preliminary findings based on the available data as of October 2012 and will 
change as recruitment continues. As such, this data is a snapshot in time. 

The survey phase provides a general picture of the perceptions, experiences, and values related to 
technology preparation and practice of K-12 teachers from the perspectives of K-12 teachers, K-12 administrators, 
and teacher education faculty members. To be able to compare and contrast between difference amounts of 
participants, percentages of participants are being used instead of the actual n values. One of the results that have 
emerged during preliminary analysis is in differences between reported teacher use, administrator perception of 
teacher use, and foci of technology activities by teacher education programs (See Figure 1). While several categories 
have clear differences such as in information presentation, administration and classroom management, analyzing 
student data, documenting growth, and higher order thinking skills, it is important to keep in mind the composition 
of the participants at this time in the study. The majority of teachers (n = 36/50) who have participated at this point 
are middle school teachers while most of the administrators who have responded are elementary principals (n = 
9/13). Discrepancies like this can skew the results significantly. Once the grade levels are more evenly represented a 
detailed and appropriate discussion will be possible. An additional area of potential skewing in results so far could 
be based on the types of school the responding teacher education faculty members are from. Of faculty members 
who have filled out any portion of the survey, most (n = 10/15) are from smaller liberal arts schools in the state. 
Once more of the larger, research schools have participated, better generalizations for the state will be possible. 

The interview phase of this study will provide richer details to help explain results from the survey and 
themes identified. The observation phase will then allow the research to document how well teacher practice aligns 
with what each group has had identified as standard practices and valuable use. 

Limitations 

A limitation of this study is that it focuses on participants from the same Southeastern state. This focus 
limits the generalizability of the findings to other states. The use of both a research record and a thick description 
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(Merriam, 2009) will help to improve the transferability of the findings to the unique contexts of each consumer of 

this research. This generalizability is also currently impeded by an uneven number of participants between types of 

participant groups in the study. 
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Introduction 

There is a call in the field of education to prepare students for the 21st century. This includes knowledge of 
21st century skills as well as the ability to adapt to and navigate a digital environment. Schools must take on the 
tremendous task of preparing students for a rapidly changing future. There is a need of a guide for educators as they 
make the transition from traditional schools to schools of the 21st century. Access to technology will play a leading 
role in this transition. However, even with access to ubiquitous computing, many schools still do not make a 
successful transformation. Giving teachers and students access to the devices of this new environment is not enough. 
Emphasis must also be put on the time, tools, community, and policy that affect the learning environment 
(Partnership for 21st Century Skills, 2009). The goal of this research was to examine the perceptions of a group of 
stakeholders as they attempted to make the transition to a school of the 21st century. It is necessary for teachers to 
continually enhance their knowledge and skills in order to prepare students for the 21st century. This includes 
keeping abreast of technological transformations (Okojie & Olinzock, 2006). Legislation requires that technology be 
incorporated in K-12 classrooms (McGrail, 2005; Zhao, 2007). For teachers to successfully prepare students and 
keep up with current legislation, they must integrate technology into their curriculum. Even though teachers are 
ultimately responsible for this technology integration, they are rarely included in the planning process (McGrail, 
2005, 2006). This research sought to give a voice to all stakeholders involved in the planning of a technology 
integration initiative. It included the perceptions of teachers, who are not typically involved in this phase. 

Review of Literature 

The review of literature for this study seeks to provide an overview of research being done on teachers’ 
perceptions of technology integration, learning environments, and the benefits and barriers of one-to-one learning 
environments. It has three sections. The first section looks at teacher’s perceptions of how pedagogy affects 
technology integration, their preparedness, and their concerns about technology integration. The second section 
examines the 21st century learning environment as compared to a traditional environment and its relation to teaching 
and learning. The third considers some of the most common benefits and barriers to implementing a one-to-one 
learning environment. 

Teacher Perceptions 

State and national legislation mandates that technology be incorporated into K-12 education (McGrail, 
2005; Zhao, 2007). The classroom teacher is ultimately responsible for ensuring the technology is integrated in the 
classroom. Unfortunately, this push for technology integration most often occurs in a top-down fashion and teachers 
are rarely included in the planning process (McGrail, 2005, 2006). It is important to survey the perceptions teachers 
have about technology integration because, as Cuban (2001) expresses, “It is the teacher, using computers or other 
information technologies as learning tools, who can, if so inclined, integrate the machines into the classroom” (p. 
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64). The extent of technology integration may be indicated by the attitudes teachers have about technology (Wang, 
Ertmer, & Newby, 2004). McGrail (2006) calls teachers the “key element in the change process” (p. 1056). That is 
why we must explore technology integration from their perspective. In this section, teachers’ perceptions of the 
affects of technology integration as related to pedagogy and the benefits and barriers of technology integration will 
be reviewed. 

Through ethnographic research, Windschitl and Sahl (2002) found that with the introduction of 
technology, teachers may change their practices over time and some adopt a student-centered approach. However, 
they note this change was not brought about by the technology, but rather the technology supported the already held 
pedagogical beliefs of the teachers - beliefs about the learners in their school, what constituted good teaching, and 
the role of technology in the lives of the students. Technology alone is not enough to be a change agent in schools. 
Teachers must hold pedagogical beliefs that support the move to a more student-centered, constructivist 
environment. These pedagogical beliefs can be supported by technology, but are not based on the presence of 
technology. In fact, the literature shows that teachers’ practices often do not change because of technology 
integration initiatives (Cuban, 2001; McGrail, 2005; Windschitl & Saul, 2002; Zhao, 2007). Rather, teachers take 
prepared lessons and try to find ways to use technology in that lesson (Okojie & Olinzock, 2006). They are not 
actually changing anything fundamental about the way they teach. 

The availability of technology and the willingness of teachers to use it are both great facilitators in the 
move towards technology integration. The majority of teachers agree that the availability of technology in schools is 
not a problem and as early as 2002, 99% of public schools reported having Internet access (National Center for 
Education Statistics, 2005). It has also been found that most teachers do not outright reject the use of technology in 
their classrooms (Cuban, 2001; McGrail, 2006). Teachers recognize many benefits to technology integration. They 
hope technology will make them more capable teachers (Zhao, 2007). 

Zhao’s 2007 study categorizes teachers’ perceived benefits of technology integration into three themes: 
efficiency, enhancement, and relaxation. Efficiency included the facilitation of paperwork and obtaining information 
that freed up more time for instructional practices. Enhancement encompasses supplementing textbooks, allowing 
for diversity in teaching strategy, and benefits to technologically competent students. Relaxation covered providing 
stimulation and motivating students as well as giving teachers a break from lecture allowing them to play the role of 
guide. 

Many teachers see the benefits technology can provide students including individualized learning 
opportunities and skill development to help them in a technological society (McGrail, 2006; Zhao, 2007). 
Technology use promotes students’ improved production and use of critical thinking. It supports student 
collaboration, interdependence, and peer support and provides variety, appeal, and motivation. (Cuban, 2001; 
Ruthven & Hennessy, 2993; Zhao, 2007). 

The top-down approach to school initiatives taken by legislators and administrators leaves teacher out of 
the planning process and causes resistance to the initiative (McGrail, 2005, 2006; Zhao, 2007). Those enforcing the 
initiative may make presumptions that are not the reality of the classroom teacher (McGrail, 2005). Some teachers 
feel the use of technology is not always appropriate for parts of or their entire curriculum. Many teacher fear losing 
their professional identity by teaching in a way that they are not comfortable or does not align with their pedagogical 
beliefs and resist moving away from instructional strategies that have worked for them (McGrail, 2005, 2006). 

Wang, Ertmer, and Newby (2004), believe that self-efficacy plays a role in willingness to integrate 
technology. Giving teachers the opportunities to increase self-efficacy (including modeling and goal-setting) can be 
difficult. Many teachers have not had formal training in technology integration (Cuban, 2001; Zhao, 2007). Too 
often, the training that is delivered is one-size-fits-all and does not meet the needs of individual teachers (McGrail, 
2005). Training must include what Windschitl and Sahl (2002) describe as “learning about” and “learning how” - 
learning about the role technology is expected to play and learning how to use the technology in the classroom. 
Teachers who have had access to quality training note positive changes in their attitudes toward technology 
integration. They report more confidence, skill, and knowledge of new ways to use technology in their classes 
(Zhao, 2007). Giving teachers resources and access to training is not enough. They also need support staff to help 
with technology problems and assist with integration (McGrail, 2005; Okojie & Olinzock, 2006). Teachers are often 
not aware of the possibilities that technology could offer their classes (McGrail, 2006). Customized training and 
expert support can help teachers find the best resources for their curriculum. 

Even those teachers who are skilled and experienced with technology have concerns about the feasibility of 
technology integration. McGrail (2005) writes, “a teacher tends to look at technology and its role in instruction from 
the perspective of a practitioner and a realist, who knows his or her own limitations a well as the constraints of his or 
her own contexts” (p. 18). Many teachers fear a disconnect between the use of technology and preparing students for 
standardized testing (McGrail, 2006; Zhao, 2007). This is a valid concern because teachers are held accountable for 
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student achievement on these tests. The introduction of technology can bring other worries to the forefront. When 
not addressed, fundamental details such as physical classroom organization can create management problems for 
teachers. Ethical concerns including copyright laws and plagiarism must be addressed (McGrail, 2005). These 
teachers must find a way to “adapt to classroom settings in which every student owns a mobile suite of powerful 
technological tools and has telecommunications access to a global repository of information and ideas” (p. 166). 
With the mounting responsibilities teachers have, many report they do not have time to attempt technology- 
integrated lessons (Cuban, 2001). 

Learning Environments 

There are certain competencies that a student must master to become successful in the 21st century. 
Partnership For 21st Century Skills (2009) has grouped these competencies into four overarching categories: Core 
subjects and 21st century themes, life and career skills, learning and innovation skills, and information, media, and 
technology skills. To propel students toward achieving these competencies, it is important to create an atmosphere 
that is conducive to acquiring these skill sets (Kennedy, 2010). This atmosphere is referred to as a 21st century 
learning environment. The Partnership for 21st Century Skills (2009) states that 21st century learning environments: 

• Create learning practices, human support and physical environments that will support the teaching 

and learning of 21st century skill outcomes 

• Support professional learning communities that enable educators to collaborate, share best 
practices and integrate 21st century skills into classroom practice 

• Enable students to learn in relevant, real world 21st century contexts (e.g., through project-based 
or other applied work) 

• Allow equitable access to quality learning tools, technologies and resources 

• Provide 21st century architectural and interior designs for group, team and individual learning 

• Support expanded community and international involvement in learning, both face-to-face and 
online, (p.5) 

While this description is all encompassing, the scope of this literature survey focused on the physical aspect 
of the environment and the support for teaching and learning this environment provides. In the following sections, a 
description of a traditional learning environment is compared to the 21st century learning environment. 

When the term “traditional learning environment” is used, it often evokes a mental picture of Industrial Age 
schools. This picture may include row-aligned desks filled with passive students who listen and take notes as the 
teacher disseminates information. Realistically, modern classrooms are a bit more flexible in nature. Desks may be 
arranged for group interaction. Technological devices such as an overhead projector and set of desktop computers 
are present. Often students can be found engaged in dialogue with each other and the teacher. Collaborative and 
individual projects are more prevalent. However, most students still rotate through their classes in a cohort 
determined by age. They attend classes for a set timeframe and earn credit only after a specified seat time or number 
of Carnegie units has been met (Partnership, 2009). 

The traditional learning environment can also be described as it relates to teaching and learning. Hannafin, 
Hannafin, Land, and Oliver (1997) briefly describe learning systems as they relate to psychological, pedagogical, 
technological, cultural, and pragmatic foundations. They posit that psychologically, early environments were 
founded in behaviorism and later in cognitivism. Pedagogically, direct instruction calls for “explicit identification of 
learning outcomes, hierarchical structures and objective-based activities, and assessment consistent with objectivist 
epistemology” (p. 104). The methods employed in traditional learning environments do not readily lend themselves 
to the development of higher order thinking skills. Astleitner (2002) asserts that overall, it is indeed “very difficult to 
successfully implement critical thinking into traditional classroom instruction” (p. 55-56). 

While the learning environment described above may have worked well for the students of the Industrial 
Age, it does not benefit today's students who will need to be able to change and adapt throughout their 
technologically changing professional careers. Brown (2006) tells us that today’s learners, facing these technological 
changes, will benefit more from “demand-pull” approach strategy - one where students learn more through 
encu Iteration and collateral learning (p. 23). This strategy will be better supported in a different type of learning 
environment - a 21st century learning environment. 

“A new electronic learning environment is replacing the linear, text-bound 
culture of conventional schools” (Cookson, 2009). The 21st century learning environment is not confined by 
physical boundaries. Learning in this environment can take place any time in any space, physical or virtual. The 
environment is made of support systems for learning (Leh, Kouba, & Davis, 2005; Partnership, 2009). 
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Physical environment. It has been noted that a 21st century learning environment is not necessarily a 
physical one; however, most schools are contained in a physical building so it is important to address what the 
environment may look like in a brick-and- mortar setting. Partnership for 21st Century Skills (2009) calls for these 
spaces to be designed with flexibility in mind. This includes moveable furniture and walls in class spaces and a 
building that will “inspire intellectual curiosity and promote social interactions” (p. 7). This flexibility in classroom 
space and furniture allows for more and better collaboration and varied teaching strategies (Harris Helm, Turckes, & 
Hinton, 2010). Students have access to resources through ubiquitous computing. This ubiquitous environment is 
obtained through the use of one-to-one portable computing devices and a wireless network infrastructure. These 
physical characteristics strengthen the teaching and learning environment. 

The learning environment includes much more than the physical descriptors. It is also comprised of time, 
tools, community, and policy (Partnership, 2009). 

lust as the physical model of a factory school is outdated, so is the strict timeframe for learning. Rather 
than relying on seat time or Carnegie units as a measure of achievement, knowledge demonstrated becomes the 
benchmark. Schools may adopt longer school hours and calendars. Informal opportunities are blended with formal 
opportunities to allow for relevant and authentic leaning and assessment (Partnership, 2009). 

The tools used for teaching and learning have also changed. In a 21st century learning environment, tools 
must be in place to support teachers and students both face-to-face and online. Technology should be in place for the 
purposes of promoting greater student achievement, increasing student engagement, assessing student performance, 
facilitating communication and collaboration, maximizing administrative effectiveness, and building student 
proficiencies in 21st century skills (Partnership, 2009). Brown (2006) asserts, “We now have the tools and resources 
for engaging in productive inquiry and learning that we never had before” (p. 20). 

Partnership for 21st Century Skills (2009) considers the most important aspect of the 21st century learning 
environment to be the “people network”. They state that “the flexible spaces that enable productive learning and 
shared work/play opportunities, the creative uses of time that promote continuous learning, the extensible 
technologies that support collaboration among the school community and the outside world - all these systems are 
valuable only in so far as they effectively support the human connections on which learning depends” (p. 21-22). 

The human aspect is important to note because often technology overshadows it when discussing 21st century 
learning environments. 

Once a 21st century learning environment has been established, it must be sustained. Sustainability can be 
achieved through policy. Educational policy covers a broad spectrum from national policy to the policy of individual 
schools. Policy will ensure that education in this new environment is “more expansive and more inclusive” 
(Partnership, 2009 p.27). 

It takes a combination of time, tools, community, and policy to create a 21st century learning environment. 
Technology can be used to support all of these features (Partnership, 2009). A one-to-one computing environment 
ensures that all faculty and students in a school have access to the digital resources they need. 

Benefits and Barriers 

Ubiquitous computing is an important part of the 21st century learning environment. However, 
implementing a one-to-one program, a program in which all teachers and students have a portable computing device 
and access to a wireless network, does not automatically ensure academic improvement (Greaves & Hayes, 2008). It 
is important to recognize there are both benefits and barriers to implementation. 

There are many benefits to implementing a one-to-one environment. The use of technology allows for a 
blend of face-to-face and online learning. A study conducted by the US Department of Education (2010) found that 
blended learning environments have a larger advantage relative to purely face-to-face instruction than purely online 
instruction. This blended environment allows learning time to take place outside of the school hours and calendar. 
Technology can facilitate collaboration and make digital resources available to all teachers and students. Cuban 
(2001) found that teachers believe technology is important for students. In his study, teachers noted technology 
allowed students more and better access to resources, a means of artifact production, and played a role in motivation. 
Perhaps the most notable benefit is the opportunity for authentic learning and assessment that ubiquitous computing 
provides. It is important for students to learn in an environment that is consistent with the world in which they will 
be expected to use the learned knowledge and skills (Harris Helm, Turckes, & Hinton, 2010; Kalny, 1999; Mouza, 
2008). These characteristics support the requirements of a 21st century learning environment. 

According to a study conducted by Project RED, properly implemented technology saves money by 
lowering copying and paperwork expenses, instructional materials costs, and lowering the dropout rate (Greaves, 
Hayes, Wilson, Gielniak, & Peterson, 2010). There are also benefits that directly impact teaching and learning. The 
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Project RED study also found that with correct implementation, the student-computer ratio impacts student 
outcomes. Access to a device increases academic achievement, improves the outcomes in intervention classes, and 
increases productivity and student engagement (Greaves, Hayes, et. al., 2010). However, it must be noted that there 
is little other research that attributes an increase in learning outcomes to technological innovations (Scrhum & 
Glassett, 2006). Cuban (2001) found that of the teachers who had successfully integrated technology into their 
classrooms, they reported lecturing less and coaching or facilitating more. 

While studies such as Project RED show overwhelming support for implementing a one-to-one program, 
they also recognize there are barriers to its implementation. The authors found that there are nine key factors to 
success of a one-to-one program. The factors are: 

• Technology is implemented into every intervention class period, 

• The principal exhibits change management leadership and provides time for teacher professional 
learning and collaboration at least monthly, 

• Students use technology daily for online collaboration, 

• Technology is integrated into core curriculum weekly or more frequently, 

• Online formative assessments are conducted at least weekly, 

• The lower the student-computer ratio, the better the outcome, 

• Virtual fieldtrips are used at least monthly, 

• Students use search engines daily, and 

• The principal is trained in teacher buy-in, best practices, and technology transformed learning 
(Greaves, Hayes, et. al., 2010). 

Through surveys, they found that only one percent of schools had implemented all nine. Finding time and resources 
to implement these factors is a barrier. 

There are many external demands on school systems and teachers that inhibit technology integration. 

Cuban (2001) provides the following insight: 

State and district requirements for graduation, age-graded organization, departmental boundaries, 
secondary teachers’ disciplinary training, and self-contained classrooms all combine to reduce cross- 
fertilization of ideas within and across departments and to encourage teachers to behave as academic 
specialists whose primary concern is covering the body of information contained within a textbook in 36 
weeks, (p. 161) 

A cost savings for schools was listed as a benefit of implementing a one-to-one environment. However, it is 
important to note that there is a substantial upfront cost that may inhibit some schools from embarking on this 
initiative (Grant, Ross, Wang, & Potter, 2005). This includes not only the cost of the devices, but also the 
infrastructure that must be in place to support their use (Speak Up, 2010). Schrum and Glassett (2006) point out the 
importance of basing ‘implementation and use on demonstrated best practices’ because of the financial investment 
(p. 2). 

Another barrier to implementation is the skill level and training of the teachers. Teachers must have the 
appropriate level of technological skill to be effective (Kalny, 1999). Their teaching practices will have to change. 
Schools will have to provide ongoing professional development that includes on-the-job experience, reflection, and 
mentoring (Greaves, Hayes, et. al., 2010; Mouza, 2008; Schrum, 1999). However, teachers report a lack of 
customized training that meets their needs (Cuban, 2001). Teachers must also become adept at combing through a 
myriad of technological resources and selecting and testing applications that are veritable and appropriate for 
classroom use (Banister, 2010; Cuban, 2001; Leh, et.al., 2005). When implementing a one-to-one environment, 
teachers will also face new classroom management challenges. Teachers must make sure students are staying on task 
while using the devices (Banister, 2010, Grant, et.al., 2005). Project Tomorrow’s 2009 Speak Up survey found that 
of teachers surveyed, 76% fear students will be distracted and 33% have concerns that students will use the devices 
to cheat (Speak Up, 2010). The International Society for Technology in Education (ISTE) developed the National 
Educational Technology Standards for Teachers (NETS*T) as a series of standards and performance indicators to 
improve the effectiveness of teachers in a blended environment (2008). Along with other organizations guidelines, 
these standards should help teachers focus their efforts on practices that will bolster learning outcomes. 

A lack of evidence of successful implementation is also a barrier. In a study of the integration of the iPod 
Touch in schools. Banister (2010) noted that a lot of the possibilities of portable computing devices have not yet 
been tested or documented. There is a lack of rigorous, generalizable research available that directly attributes 
learning outcomes to technology integration (Leh, et.al., 2005; Schrum and Glassett, 2006; US Department of 
Education, 2010). 
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Chapter Summary 

It is important to include K-12 teachers in the planning phase of a technology integration initiative 
(McGrail, 2005, 2006). The responsibility for successful integration rests with them. Their pedagogical beliefs, 
attitudes, and abilities are factors in the success of the initiative. The addition of technology does not change the 
learning strategies utilized by teachers (Cuban, 2001; McGrail, 2005; Windschitl & Saul, 2002; Zhao, 2007). 
Technology will, however, provide support for those teachers who believe in a student-centered, constructivist 
approach to learning (Cuban, 2001). Technology can support the administrative duties of teachers as well as enhance 
the text and lessons. For students, it provides the opportunity for individualized learning, collaboration, critical 
thinking, and artifact production (Cuban, 2001; Ruthven & Hennessy, 2993; Zhao, 2007). There are obstacles to 
successful integration. Time and customizable training are not readily available to teachers (Cuban, 2001; McGrail, 
2005; Zhao, 2007). Also, the demands on teachers cover the curriculum in a finite time period and to prepare 
students for standardized testing does not easily lend itself to the introduction of technology-integrated lessons 
(McGrail, 2006, Zhao, 2007). 

A 21st century learning environment supports learners as they obtain the skills necessary for success. This 
environment is comprised of time, tools, community, and policy (Partnership, 2009). Technology, and specifically 
access to ubiquitous computing, supports all aspects of the 21st century learning environment. This environment 
allows for more flexibility and authenticity in education and assessment. It gives teachers and students equal access 
to resources. One-to-one environments also promote academic achievement and lower costs for schools 
(Partnership, 2009). However, there are barriers to implementation. Finding time and resources to support full 
implementation can be burdensome for schools. There is also a substantial up-front cost for hardware, infrastructure, 
and training and professional development (Grant, et.al., 2005; Speak Up, 2010). Teachers must also be trained to 
use technology appropriately for teaching and learning (Kalny, 1999). Perhaps the largest barrier is a lack of 
rigorous research or models of successful implementation for schools to follow. (Banister, 2010; Leh, et.al., 2005; 
Schrum and Glassett, 2006; US Department of Education, 2010). 

Methodology 

The purpose of this study was to capture what the members of a steering committee believe is the vision 
and purpose of the one-to-one initiative being proposed at a private. Catholic, all boys high school; what they 
thought teaching and learning in the new environment would look like and what they think it looks like now; what 
are their ideas of potential benefits and barriers of the change; and what they believe are the critical components for 
this initiative to be successful at the school. This study was guided by three research questions: 

1. What are the committee members’ perceptions of a one-to-one environment? 

2. How will the one-to-one environment compare to the current environment? 

3. What are the benefits and barriers of implementing the one-to-one initiative? 

Research Design 

This study was developed using qualitative research. According to Creswell, qualitative research is “an 
inquiry process of understanding based on a distinct methodological tradition of inquiry that explores a social or 
human problem” (p. 249). The perceptions of a steering committee comprised of school faculty and staff, 
administration, and community members and their views of implementing a one-to-one initiative were explored. A 
qualitative methodology can best capture the views of the participants in their environment. It allowed their voices 
to be heard and their authentic perceptions to be explored. 

The approach to inquiry used in this study was the case study. A case study is the study of a bounded 
system or case “over time through detailed, in-depth data collection involving multiple sources of information” 
(Creswell, 2007, p. 73). The boundaries of this case were the site, the time period, and the steering committee to 
which the participants in the study belong. The components of this case study including the site of research, 
participants, methods of data collection and analysis, risks and benefits, and ethical considerations are discussed in 
the following sections. 
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Site of Research 


This study was conducted at a private, all-boys high school in a city with over 100,000 inhabitants. We will 
refer to this school as St. John’s. St. John’s has a college preparatory focus and serves 850 students in grades nine 
through twelve. The students have diverse social, cultural, and economic backgrounds. There are 70 faculty and staff 
members and four administrators. The student to teacher ratio is twelve to one. Teachers at St. John's have been 
teaching for an average of 15 years and an average of 13 years at St. John's. 

Participants 

The school principal selected the one-to-one initiative steering committee participants. The committee 
includes 15 representatives from various groups including administration, faculty, technology support staff, alumni, 
and community members. A cross section of this committee was selected for data collection. 

Stratified purposeful sampling was used to select participants because, according to Miles and Huberman, it 
“illustrates subgroups and facilitates comparisons” (as cited in Cresswell, 2007, p. 127). One administrator, two 
classroom teachers, one technology support staff member, and one community member/alumnus represent the 
subgroups in this study. A stratified sample allowed for comparisons in perceptions to be made across the 
subgroups. 

Data Collection 

In-depth, individual, face-to-face interviews were used to collect data for this study. These interviews 
ensured that the participants own words were used and allowed the interviewer to seek clarity from the participant. 
Each interview question was aligned to one of the three research questions for this study. Voice recording and 
member checks were used to affirm that the participants’ perceptions have been accurately captured. 

Data Analysis 

After each interview was conducted, the digital voice recording was used to make a transcript of the 
interview. Each transcript was read through twice before the consecutive interview was conducted. The purpose of 
the preliminary readings was to identify new veins of insight that needed to be addressed or areas that needed more 
clarification from participants. Additional questions were then added to the interview protocol based on findings 
from the preliminary readings. After all five interviews were conducted, the transcripts were analyzed. Each 
transcript was read through in its entirety and all key points made by participants were highlighted. These key 
points were then transferred to sheets of color-coded paper. The sheets of colored paper where then read through 
twice and themes from these initial points began to emerge. The emergent themes were then transferred to individual 
note cards. Then, this entire process was repeated. Overarching themes were then identified and written on note 
cards. The themes from the note cards were inserted into a table. The table contained rows for each theme and 
columns that corresponded to each participant. Then the original transcripts were re-read and evidence of each 
theme was identified. The evidence was comprised of examples and quotes. These examples and quotes were 
inserted into a corresponding cell on the table. The theme supporting evidence from each participant was then 
compared. Each theme and supporting evidence from the transcript was recorded. This information was sent to each 
participant via email. Participants were asked to review for accuracy and given the opportunity to ask questions or 
provide clarification. Data was then aligned with each research question. 

Report of the Findings 

Four overarching themes emerged during this study: Relevance in the 21st century, change in the teaching 
and learning environment, a need for leadership, and support for implementation. An interpretation of each theme is 
presented. 

Relevance in the 21st Century 

All participants of this study agreed that a desire to remain relevant in the 21st century to be the purpose of 
the one-to-one initiative at St. John's. This theme of relevance included two sub-themes: Authentic learning and 
technology skills. 
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Participants recognized the importance of their students making a connection between school and life. This 
observation is consistent with the literature that it is imperative for students to learn in an environment that is 
consistent with the world in which they will be expected to use the learned knowledge and skills (Harris Helm, 
Turckes, & Hinton, 2010; Kalny, 1999; Mouza, 2008). Participants noted that gaining technology skills would be a 
benefit for the students. These skills would help the students to adapt to a changing work environment. The benefit 
of skill development was also noted in the literature (McGrail, 2006; Zhao, 2007). 

St. John's has a college preparatory focus and touts an academic program that prepares young men for 
college and careers. Therefore, it is important that the school prepares its students to meet the requirements of the 
21st century. It is important to note, as one participant did. that St. John's is also a school steeped in tradition. It is 
possible that the desire to keep to tradition may serve as a point of contention when moving forward with the 
changes a one-to-one environment will bring. 

Change in the Teaching and Learning Environment 

This theme encompassed a comparison of what the participants believed was the current environment at St. 
John's to a one-to-one environment. It included three sub-themes: Teacher actions, student actions, and curriculum. 

All participants noted that currently teachers at St. John’s deliver content through lecture. With few 
exceptions, the learning environment is teacher centered. Most participants felt that this would change with the one- 
to-one initiative. They believed the computers would facilitate a more student-centered environment. Instead of the 
current situation where the teacher lectures and the student takes notes or participates in question and answer 
sessions, the new environment would include an inverted or “flipped” approach. In this approach, content would be 
delivered before class using the portable learning device. Class time would then be used for discussion, 
collaboration, and higher order thinking activities. While the participants’ views concerning computers facilitating a 
more student-centered environment were in line with the literature, there is also a disconnect. Through the review of 
previous studies, we know that the introduction of technology is not enough to be a change agent in schools. 
Teachers must hold pedagogical beliefs that support the move to a more student-centered, constructivist 
environment. These pedagogical beliefs can be supported by technology, but are not based on the presence of 
technology. The literature shows that teachers' practices often do not change because of technology integration 
initiatives (Cuban, 2001; McGrail, 2005; Windschitl & Saul, 2002; Zhao, 2007). 

Anna, the technology staff member, also raised the concern that the teachers had a responsibility to prepare 
the students for standardized tests and college entrance requirements. She was unsure how teachers would find the 
time to properly integrate technology and still make sure the students were prepared for these exams and college 
entrance. Her concerns were validated by the literature (Cuban, 2001; McGrail, 2006; Zhao, 2007). 

Participants also expressed concerns about classroom management and ethics. They expressed fears that 
students would abuse the machines, be more inclined to cheat, or have trouble staying on task. These concerns were 
validated by the literature (Banister, 2010, Grant, et.al., 2005, Speak Up, 2010). 

A Need for Leadership 

Participants expressed a need for leadership as St. John's made the transition to a one-to-one school. This 
theme included three sub-themes: Planning, buy-in, and communication. 

The literature tells us that the top-down approach to technology integration initiatives is often met with 
resistance by teachers (McGrail, 2005, 2006; Zhao, 2007). The participants in this study also expressed that 
sentiment. It is interesting to note that conflicting views of the administrator and the faculty and staff members on 
the committee. While the administrator was able to articulate a vision and plan for the initiative, the faculty and staff 
members did not express awareness of many of the details. The administration has included teachers in the planning 
process; however, they were not a part of the initial decision to become a one-to-one school. The administrator 
believes teachers are included because they have been asked to serve on the steering committee and sub-committees. 
However, through the interviews, it became evident that some teachers still feel this is being forced on them. This 
phenomenon is not surprising because it has been documented in the literature (Greaves, Hayes, et. al., 2010; 
McGrail, 2005, 2006). 
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Support for Implementation 

This theme encompassed what participants perceived to be the critical components that had to be in place 
for the one-to-one initiative to be a success. It included two sub-themes: Support of teachers and support for 
infrastructure. 

All participants noted that support and professional development for teachers was of utmost importance. It 
was also mentioned that this support consisted of more than just training. It had to also include mentoring and 
support staff. The literature also holds this view (Cuban, 2001; Greaves, Hayes, et. al., 2010; Kalny, 1999; McGrail, 
2005; Mouza 2008; Okojie & Olinzock, 2006; Schrum, 1999; Windschitl & Sahl, 2002; Zhao, 2007.) 

All participants expressed a need for the infrastructure to be of the highest quality and thoroughly tested 
before a full implementation takes place. This would include testing the devices, network, learning and content 
management system, and any software. It also includes a pilot with groups of teachers and students. Hiring 
additional support for the infrastructure was also a concern. Paying special attention to the infrastructure was also a 
concern found in the literature (Cuban, 2001; McGrail, 2005; Okojie & Olinzock, 2006). 

Conclusion 

We have entered an era in education where we must equip students with the skills to be successful in the 
21st century. This includes the skills to adapt to and navigate a digital environment. St. John's has answered this call 
by preparing to become a one-to-one school - a school in which all teachers and students are equipped with a 
personal computing device and high-speed Internet access. 

A review of current literature has informed us that the 21st century learning environment is different from 
the traditional environment. The steering committee at St. John's recognizes and is planning for these differences. 
They have identified many of the same barriers as other schools that have implemented one-to-one environments. 
They also hope to realize the potential benefits this environment may afford. 

The literature also informed us that teachers who do not already hold pedagogical beliefs that support a 
student-centered, constructivist learning environment will most likely not change their teaching practices. The 
administrators, faculty, and support staff at St. John's as well as those learning from St. John's example, should 
examine this closely and seek out strategies to help with this transition. 

Many schools have implemented one-to-one programs. It was a surprise to me how closely the experiences 
and perceptions of the participants in this study mirrored those from other studies. It is affirming to know the 
concerns they have and struggles they face are not unique. The findings offer a great opportunity for St. John’s to 
align their plans with those of schools who have had success. It is still early enough in their efforts to correct 
mistakes and misconceptions and build on their strengths. 

There is still much research to be conducted in the area of technology integration. Technology is constantly 
changing and educators will have to continually work to stay abreast of new innovations that can sustain, enhance, 
or transform their instructional strategies. The requirements of the 21st century call for an educational system that 
prepares students for a technologically enhanced, fast-paced, and dynamic work environment. Continued research 
will enable educators to be better prepared to meet the needs of these students. 
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Abstract 

Collaborative learning with new technologies to engage students in their learning materials and learning 
activities has been one of the new trends in education settings. Among which, the study of incorporation of wikis, 
one of the Web 2.0 technologies for both online and hybrid courses to promoting collaboration among students, has 
been the main focus of many educators recently (Auger et al., 2004; Bergin, 2002, Slotter; 2010; Mindel & Verma, 
2006). This study examined the effectiveness of the Wiki tool in Blackboard 9.1 course management system. The 
results indicated that pre-group activities are necessary to help students get started with their Wiki group project and 
Blackboard Wikis are effective in promoting online collaboration. 

Introduction 

Collaborative learning with new technologies to engage students in their learning materials and learning 
activities has been one of the new trends in education settings. Among which, the study of incorporation of wikis, 
one of the web 2.0 technologies for both online and hybrid courses to promoting collaboration among students, has 
been the main focus of many educators recently (Auger et al., 2004; Bergin, 2002, Slotter; 2010; Mindel & Verma, 
2006). Results show that wikis can be used for a variety of tasks (Mindel & Verma, 2006); the use of wikis could 
motivate students, enhance their enjoyment and engagement in the learning activities, support constructive learning 
model (Slotter, 2010; Mindel & Verma, 2006), and Wikis have been considered “the easiest and most effective web- 
based collaboration tool” (Educause, 2005). 

However, the studies of the usefulness of wikis so far mainly concentrated on public wiki software such as 
Wikipace and PBwiki; few studies have explored the use of wiki tools specifically designed within a learning 
management system, such as Blackboard 9.1. While the wiki tool in Blackboard 9.1 bears some similar functions 
such as allowing students to write, edit, and modify the content collaboratively. Blackboard 9. 1 wiki tool also allows 
the instructor to view contributions of each student and assign grades easily. Therefore, it is necessary to investigate 
how the Wikis in BB9.1 can enhance student learning. 

Web 2.0 Learning Environments and Wikis 

Web2.0 technologies refer to web applications that can facilitate participatory information sharing and 
collaboration on the Web. In a web2.0 learning environment, learners and learners, learners and instructor are 
closely connected via cyber world. Participants in the learning environment interact with each other and play 
different roles to form social networks and build a learning community (Tu, Blocker & Roberts, 2008). Based on Tu, 
Blocker & Roberts’ (2008) theattical metaphor for web 2.0 learning environments, there are four dimensions that 
constitute a web2.0 learning environment: Cognitive, social, networking and interaction. The cognitive dimension 
focuses on the process of individual thinking about how they can engage and contribute in their learning. The social 
dimension refers to “actors (learners/instructors) and their relationships to others” (Tu, Blocker & Roberts, 2008. p. 
258). The network dimension refers to network technology stage that empowers learners to use various advanced 
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technology tools to learn. The integration dimension refers to the aspect that learners are engaged in or get involved 
in learning activities in various contexts (Tu, Blocker & Roberts, 2008). 

Wiki as an advanced web2.0 tool can enhance learning in all these four dimensions. The word "wiki" 
means quick or fast in Hawaiian. It was first adopted by Ward Cunning in 1994 for a collaborative tool that he 
developed for use online ( Augar, Reitman& Zhou, 2004). A wiki is a website that allows collective work of many 
authors to write, edit, delete or modify content that has been placed on the web site using a browser interface (Toki, 
2005). As one of web2.0 learning environments, Wikis allow multiple users to engage in collaborative learning. 
Collaborative learning means “joint intellectual effort by students, or students and teachers together”. In 
collaborative learning, students work in small groups, exploring course materials, searching for understanding, 
solutions, or meanings, or creating a product, etc. Researchers show that students tend to learn more and retain 
longer of the course materials when working in small groups and feel more satisfied with their classes (Davis, 1993). 

Tu, Blocker & Roberts (2008) insist that, in cognitive dimension, wikis technology allow the learners to get 
engaged in process of sharing, building, exchanging building and modifying their mental modeling by interacting 
with other learners and gain their identities. Interaction in wikis also create an interaction space that allows learners 
to negotiate meaning, build knowledge and empower collaborative learning activity and thus, support the social, 
network and integration dimensions. Mark & Conian (2008) integrated wiki in their ESL writing class and found out 
that the use of wiki enhance student’s creativity in writing and improve student wring quality. Slotter (2010) uses 
Wiki for collaborative learning in his social psychology course for final project with a great success. There are many 
ways to incorporate wiki in various course subjects for different activities. Mindel and Verma (2006, p. 8) 
summarize the following wiki activities that can be used for student collaboration: 

• Application of concepts to real-world scenarios 

• Business and technology analyses 

• Frequently Asked Questions (FAQs) [Bergin, 2002] 

• Glossary [Schneider, 2003] 

• Hints on how to use the course software effectively [Bergin, 2002] 

• Home pages for students [Bergin, 2002] 

• Icebreaker exercise [Auger et al., 2004] 

• Knowledge management system experience [Raman et al., 2005] 

• Fiterature review 

• Fiving encyclopedia about university or university community5 

• Peer review of writing exercise 

• Personal portfolios [Higdon, 2006] 

• Signup sheet [Gill, 2005] 

• Student journaling [Higdon, 2006] 

• Syndicating / Aggregating web resources [Higdon, 2006] 

On Smart Teaching website, there as many as 50 ways of using wikis are suggested, ranging from resource 
creation to student participation, group projects, student interaction, classroom space creation and community 
building etc. There is no doubt that wikis can be used as a powerful tool to get students involved in engaging, 
meaningful and constructivist learning. 

Wikis in Blackboard 9.1 

In BB9.1, instructors can create two types of wikis for students in the class: Course Wikis or group Wikis. 
Course Wikis are for the entire class to contribute; while Group Wikis are for the members in the groups to 
collaborate. Both types allow students to write, delete and modify one or more pages of using WYSIWIG (What 
You See Is What You Get) text editor. Instructors can create multiple wikis that students can build their own “web” 
inside of Blackboard courses. 

Wikis in Blackboard provide a unique feature that allows instructor to view a participation summary of 
Course wikis or group wikis. The participation summary provides the number of words modified by each student 
and the number and percentage of page contribute by each student. It also displays the history of the wikis and 
allows instructor to compare the different versions and retrieve information about the development and contributions 
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for any individual over time A other unique feature for wikis in Blackboard 9.1 is that instructor can assign a Wiki 
to be grade or not. Once a Wiki is set to be graded, a grade column is automatically created in the Grade Center. 

Overall, the wikis tool in Blackboard 9.1 seems to be more powerful and flexible for instructors to create 
different types of wiki activities. It is more effective and much easier in evaluating or assessing students’ 
contribution and achievements for their collaborative learning activites. It is believe that this paper would provide 
more evidence or insights on whether and these unique features support students’ collaborative learning in sharing 
knowledge, exchanging ideas, organize information and engage in learning in this CMS wiki learning environment. 

The study 

This case study examined the effectiveness of the Wiki tool in a course management system in working on 
group projects in spring 2011. Students in two undergraduate courses in a Midwest university formed their own 
groups to work on a project by utilizing the Wiki tool in Blackboard 9. 1. To help students get started, they were first 
assigned to work on simple projects in temporary groups in class and online. Then they formed their own groups 
with four to five members each to work on a course related project. A survey with 19 questions was constructed 
based on current research. Data were collected to answer the research questions regarding the effectiveness of the 
activities used to prepare students to collaborate on a Wiki project, and the effectiveness of the Wiki tool of 
Blackboard 9.1. Specifically, this paper addressed the following questions: 

1. What activities are useful to help students get started before working on a collaborative Wiki project? 

2. Is Blackboard 9.1 Wiki tool effective in collaborative learning? 

3. How can the instructor effectively use Wikis to promote collaborative learning? 

Two cases 

Wikis were used for students in two undergraduate courses to collaborate on small group projects. The first 
course is about using technology and multimedia in educational settings. Students who take the course are mainly 
undergraduate pre-service teachers, who will teach in K-12 schools in the future. The course covered various topics 
in using technology in the classroom and a variety of software applications such as MS Word, PowerPoint, Excel, 
Inspiration, Audacity, Movie Maker, Google Sites, and Google Earth. It was taught in a blended format, in which 
students had opportunities to work in the lab on projects and plenty of opportunities to use Web2.0 technologies to 
share their ideas regarding issues in using technology in the classroom. One unique project was the Wiki group 
project. Students formed their own groups of 3-4 based on their interest in a topic that was relevant to technology 
use in education and used the Wiki in the Blackboard course site to create, edit, and finalize their projects. They 
were required to use the group Wiki in the Blackboard course site to make changes of the project. However, they 
were allowed to use other tools such as email and discussions to exchange ideas. The project was graded based on a 
set of rubrics. Individual student’ contributions were taken into consideration for students' final grades. Nine of the 
24 students in this class responded to the survey posted on Surveymonkey.com. 

The second course is a web design course for undergraduate students majoring in business management. 
The course covered various topics in web design for business communication. Students learned to use Dreamweaver 
to construct their individual and small group projects. The Wiki project for each group was to create a web design 
and development plan to address the critical aspects of the web design process from design to publish and 
maintenance. Students were required to use the course Wiki tool to construct their group project. The project was 
graded based on a set of questions regarding the web design process and the objectives set by each group. Individual 
student’ contributions were taken into consideration for students’ final grades. 

A total of 18 students in these two classes (n=37) responded to the survey, in which 9 out of 24 responded 
from Educational technology class, and 9 out of 13 responded from the Web Design course. The response rate was 
49%. The demographics were illustrated in Tables 1. 
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Results 


Research question #1: What activities are useful to help students get started before working on a collaborative Wiki 
project? 

Students reported that pre -group activities including the online and in class group activities, online video tutorials, 
and the instructor demonstration in the first few weeks were necessary to help them get started in forming their 
groups, leaning how to use Wiki tools. Table 2 provides the mean and standard deviation for each of the pre-group 
activities evaluated in the study. Tables 3-5 provide frequency and percentage on each activity by rating from 
strongly disagree (SD) to strongly agree (SA). 

Table 1 


Student Demographic Summary 



Frequency 

Percentage 

Course taken 

Educational Tech 

9 

50 

Web Design 

9 

50 

Total 

18 

100 

Status 

Full time 

16 

89 

Part time 

2 

11 

Total 

18 

100 

Year in College 

Freshmen 

3 

17 

Sophomores 

3 

17 

Juniors 

6 

34 

Seniors 

6 

34 

Total 

18 

100 
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Table 2 

Individual Mean Scores for Each Pre-Group Activity 


Activities 

N 

Minimum 

Maximum 

Mean 

Std. Deviation 

Pie small-group activities 

18 

3 

5 

3.94 

.639 

Video tutorials 

18 

2 

5 

3.39 

.916 

Instructor demonstration 

18 

2 

5 

3.61 

.850 

Valid N (listwise) 

18 






Table 3 

Frequency and Percentage on Pre-small Group Activities 



Frequency 

Percent 

Valid Percent 

Cumulative 

Percent 

Valid N 

4 

22.2 

22.2 

22.2 

A 

11 

61.1 

61.1 

83.3 

SA 

3 

16.7 

16.7 

100.0 

Total 

18 

100.0 

100.0 



Table 4 

Frequency and Percentage on Video Tutorials 



Frequency 

Percent 

Valid Percent 

Cumulative 

Percent 

Valid D 

4 

22.2 

22.2 

22.2 

N 

4 

22.2 

22.2 

44.4 

A 

9 

50.0 

50.0 

94.4 

SA 

1 

5.6 

5.6 

100.0 

Total 

18 

100.0 

100.0 



Table 5 

Frequency and Percentage on Instructor Demonstration 



Frequency 

Percent 

Valid Percent 

Cumulative 

Percent 

Valid D 

2 

11.1 

11.1 

11.1 

N 

5 

27.8 

27.8 

38.9 

A 

9 

50.0 

50.0 

88.9 

SA 

2 

11.1 

11.1 

100.0 

Total 

18 

100.0 

100.0 
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The open-ended question in the first part of the survey solicited other activities that instructors can use to help 
students get started with their Wiki group projects. Eight participants (44.4%) offered comments and suggestions. 
The responses to the open-ended question revealed that: 

• Wikis are self-explanatory and easy to use. 

• The demonstrations and videos are necessary and helpful. 

• At least one work day is required to learn how to use Wikis on group projects. 

Only one student indicated that s/he would rather collaborate in real life than using Wikis online. 

Research question #2: Is Blackboard 9.1 Wiki tool effective in collaborative learning? 

Students reported that Blackboard Wikis helped them to create, revise, share knowledge in their groups. They also 
reported that the Wikis helped them track their contributions to the group projects and the tracking features 
motivated them to contribute to the projects. Table 6 provides the mean and standard deviation for each of functions 
and features of Blackboard Wikis evaluated in the study. 

Table 6 

Individual Mean Scores for Effectiveness of Wikis 



N 

Minimum 

Maximum 

Mean 

Std. Deviation 

Create 

18 

3 

5 

4.00 

.594 

Revise 

18 

2 

5 

4.00 

.840 

Share 

18 

2 

5 

4.11 

.676 

Track 

18 

2 

5 

4.06 

.639 

Motivation 

18 

3 

5 

4.11 

.583 

Learned By Contributing 

18 

2 

5 

3.78 

.878 

Learned From Others 

17 

1 

5 

3.47 

1.007 

General Effectiveness 

18 

2 

5 

3.56 

.984 

Valid N (listwise) 

17 






When asked whether they would like to use the course wikis or other software for a future group project students 
66.7% of the participants would use the course Wiki tools, 5.6% would use other Wiki tools such as Wikispot, and 
27.8 % would rather use Word and PowerPoint for group projects. 

The open-ended question in the first part of the survey solicited other activities that instructors can use to 
help students get started with their Wiki group projects. Four participants (22.2%) offered positive comments and 
suggestions. The responses to the open-ended question revealed that the course wikis are a great tool for group 
members to collaborate on team projects and keep track of their contributions to the final project. However, one 
student stated that “it's very helpful in collaboration, but better software would make the experience that much 
better. I especially dislike the html feature where you have to toggle back and forth to edit things and such. It's 
irritating.” Another student commented that some groups had much more experience than others so work will be 
skewed and they should not be penalized for that. Therefore, instructors will need to help students become 
comfortable with the Wiki tool in the beginning of the semester. 

Research question #3: How can the instructor effectively use Wikis to promote collaborative learning? 

Based on the students’ responses, the instructor can help use the following strategies when using course 
Wikis to promote collaborative learning: 

• Use easy small group activities to build student confidence in contributing to group projects. 

• Give students a work day in the classroom to learn to use Wikis for online collaboration. 

• Provide online tutorials for students to refer to throughout the semester. 

• Instructors should be available in class to answer any questions about Wiki group projects. 
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Discussions 


The findings of this study indicated that Blackboard course Wiki tool is effective and easy to use. However, 
not all assignments can be conducted effectively with Wikis. Three critical factors need to be given consideration 
when adopting Wikis, types of assignments, student readiness, and on-going support in the group process. 


Types of Assignment 

Collaboration is not easy, so not all assignments are worth of the time and effort for students to collaborate. 
Mandernach (2010) identified three types of assignments that are suitable to group work (1) there’s no right answer, 
such as debates, or research on controversial issues; (2) there are multiple perspectives, such as analyzing current 
events, cultural comparisons, or case studies; (3) there are too many resources for one person to evaluate, so a jigsaw 
puzzle approach is needed with each student responsible for one part. 

Student Readiness 

Whether in an online course or a hybrid course, when using a new technology, student’s readiness is the 
key to a successful learning experience. This is no exception for using wiki tools. Therefore, it is very important for 
the instructor to prepare students both in getting familiar the wiki tool and help them understand their roles in the 
group project. Instructor should communicate well about his/her expectation on how she/he expects the group 
members to work together and how he would evaluate individual student’s contribution. It is indicated in this study 
that instructor demo, online tutorials and pre-small group learning activities, can greatly help build students’ 
confidence and reduce anxiety in using the new tool. 


On-going Support 

Comparing to other collaboration tools. Wikis are featured with easy publishing capabilities (Parker & 
Chao, 2007). Blackboard course Wiki tool is an integrated part of other course components such as lectures, 
communications, assessment etc. It makes collaborative work easy to manage. Students can create, edit, and track 
their work in one single course site. Instructors can also keep track of each member’s contributions to group projects 
by viewing statistics. On-going support from the institutional help desk is helpful to students when they encounter 
technical problems. 


Conclusions 

Students reported that Blackboard Wikis were helpful in online collaboration, especially when working on 
small group projects. Pre-group activities such as pre-small group activities, online tutorials, and instructor 
demonstration were necessary and helpful. It was also recommended that a work day in class be planned on how to 
collaborate online by using Wikis. Instructors need to plan these activities and make them available to students 
before a Wiki group project is assigned. 
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